Expensive dot-sourcing in PowerShell modules, revisited

A PowerShell module I use has 200+ functions split into a single file each and about 17kloc. It was taking about 30s to load so I started looking for ways to shave time off.

If you've done the same then you'll know few years ago Bartek Bielawsk wrote a post about Expensive dot-sourcing. The gist of it is as follows:

Dot sourcing individual files is slow.
Using .NET functions to read the file first is faster.

One of the more heated ideas is that you should combine all of your scripts into a massive file and execute it once instead. Currently dbatools uses a mix of these techniques in great detail and to great success.

I don't want to use a single file because I don't like build the idea of a pre-compile build pipeline for a scripting language, so here's an alternative and benchmarks of how these techniques stack up (taken on Windows 10 PS 5.1 obviously your results may differ).

First let's create a dummy PowerShell module and populate it with a lot of garbage functions doing garbage things.

if (!(Test-Path C:\Temp\LoadTest)) {
    Remove-Item C:\Temp\LoadTest -Recurse -Force
}
New-Item C:\Temp\LoadTest -ItemType Directory
New-ModuleManifest C:\Temp\LoadTest\LoadTest.psd1 -RootModule "LoadTest.psm1"
1..300 | ForEach-Object {
    $functionName = "Function_$(([string] $_).PadLeft(3, "0"))"
    Set-Content -Path "C:\Temp\LoadTest\$($functionName).ps1" -Value ("
function $functionName {
    [CmdletBinding()]
    param (
        [Parameter(ValueFromPipeline)]
        `$A,
        [Parameter(ValueFromPipelineByPropertyName)]
        `$B
    )

    begin {

    }

    process {
" + (1..300 | ForEach-Object { "
        if (`$A -eq `$B) {
            `$A
        } else {
            `$B
        }
"} ) + "
    }

    end {

    }
}
")
}

Now we can start testing the different load techniques. We do this in a new PowerShell session each time because it seems some caching is going on and skewing results otherwise.

Here's dot sourcing each file. It averages 5.4s for me which actually is pretty good. Maybe my dummy function code isn't complex enough to stress the parser!

Set-Content -Path "C:\Temp\LoadTest\LoadTest.psm1" -Value 'Get-ChildItem $PSScriptRoot *.ps1 | ForEach-Object {
    . $_.FullName
}'
(1..10 | ForEach-Object { &powershell { (Measure-Command { Import-Module C:\Temp\LoadTest -Force }).TotalSeconds } } | Measure-Object -Average).Average

Reading the files using .NET cuts this down to 4.0s for me.

Set-Content -Path "C:\Temp\LoadTest\LoadTest.psm1" -Value 'Get-ChildItem $PSScriptRoot *.ps1 | ForEach-Object {
    . ([scriptblock]::Create([System.IO.File]::ReadAllText($_.FullName, [Text.Encoding]::UTF8)))
}'
(1..10 | ForEach-Object { &powershell { (Measure-Command { Import-Module C:\Temp\LoadTest -Force }).TotalSeconds } } | Measure-Object -Average).Average

The suggested practice is to roll all the files into one and for this test we'll just write it directly into the psm1. This takes us down to 2.0s!

$content = Get-ChildItem C:\Temp\LoadTest *.ps1 | ForEach-Object {
    [System.IO.File]::ReadAllText($_.FullName, [Text.Encoding]::UTF8) + [Environment]::NewLine
}
Set-Content -Path "C:\Temp\LoadTest\LoadTest.psm1" -Value $content
(1..10 | ForEach-Object { &powershell { (Measure-Command { Import-Module C:\Temp\LoadTest -Force }).TotalSeconds } } | Measure-Object -Average).Average

My theory was that it's the parser that takes all the time rather than reading the files, and so we could skip the publish step which creates a large cache file, and just read the files normally while only invoking the parser once. It takes 1.7s. I don't know why but it's at least on par with a cache file.

Set-Content -Path "C:\Temp\LoadTest\LoadTest.psm1" -Value '$content = Get-ChildItem C:\Temp\LoadTest *.ps1 | ForEach-Object {
    [System.IO.File]::ReadAllText($_.FullName, [Text.Encoding]::UTF8) + [Environment]::NewLine
}
. ([scriptblock]::Create($content))
'
(1..10 | ForEach-Object { &powershell { (Measure-Command { Import-Module C:\Temp\LoadTest -Force }).TotalSeconds } } | Measure-Object -Average).Average

The caveats with these fast loading processes are:

It doesn't work with DSC resources.
It doesn't work well with PowerShell 5 classes.
It doesn't work well with debuggers, so you can include a boolean like in Bartek's article, check for a .git folder like dbatools, or even check for a VS Code host process and use the fast or slow technique accordingly.

But that leaves a lot of room to improve your modules. Give one a try.

Share on: