You may (not) have seen the blog today where current VMware Tools on Windows have yet another a broken receive-side scaling implementation causing random network failures. The good news is that it's not enabled by default.
But what if you have thorough VMware admins who enable cool features? Or if you have a large environment with hundreds of thousands of random daily connectivity ring buffer errors and timeouts, and want to cross this off of the never-ending list of possible causes? Or… maybe you just want to make sure you never, ever experience this problem on any legacy servers you later acquire?
Normally you could just test for the latest version of the tools on your servers but the fixed version isn't out yet. We're going to need to do this the hard way.
What we're looking for is whether the driver has RSS enabled. (Note: Yes, this shows Intel instead of VMXNET3, read on regardless).
You might read online that it's impossible but that's nonsense. There's almost always a way if you look hard enough, especially for a very specific issue like this.
First install my Cim module unless you enjoy writing reams of pointless code, as we're going to use it to manage Cim connections and do fast remote registry queries (even to Windows Server 2003). I'm using PowerShell 5 as this will make everything easier.
Install-Module Cim
We need two pieces of information to gather this RSS setting. There's a Device ID which is a four digit number that represents which network adapter this is, and there's also a GUID which is used for all network adapters on the system and which is hardcoded by Microsoft (though you can also query it from the registry from Win32_PnPSignedDriver).
To filter the network adapters you could focus on a manufacturer of "VMware, Inc." but for my demo machine which runs on VMware Fusion it uses Intel drivers so I'll grab those too.
$InputObject = $env:COMPUTERNAME
$cimSession = New-CimSessionDown $InputObject
$networkAdapters = Get-CimInstance -OperationTimeoutSec 30 -CimSession $cimSession -Class Win32_NetworkAdapter -Filter "Manufacturer = 'VMware, Inc.' Or Manufacturer = 'Intel Corporation'"
foreach ($networkAdapter in $networkAdapters) {
We need to query a registry key now based on Device ID.
$setting = Get-CimRegValue -CimSession $cimSession -Key "SYSTEM\CurrentControlSet\Control\Class\{4d36e972-e325-11ce-bfc1-08002be10318}\$($networkAdapter.DeviceID.PadLeft(4, '0'))" -Value "*RSS" | Select-Object -ExpandProperty Data
That's it. If $setting -eq 1 then we have RSS enabled, otherwise it doesn't exist or it's disabled.
There's a second part though which is working out what version of VMware Tools we have on the system to know whether it's in the affected version range from the blog post. There may be a better way to do this but I found it possible by querying the service, getting the file version of the executable, and using that.
$imagePath = Get-CimRegValue -CimSession $cimSession -Key "SYSTEM\CurrentcontrolSet\Services\VMTools" -Value ImagePath | Select-Object -ExpandProperty Data
if ($imagePath -match '"(.*)"') {
$imagePath = $Matches[1]
}
$imagePath = "\\$InputObject\$($imagePath.Replace(':', '$'))"
$image = Get-Item $imagePath
if ($image.VersionInfo.FileVersionRaw -ge [version] "9.10" -and $image.VersionInfo.FileVersionRaw -lt [version] "10.1.6") {
Easy. (Note: I check for less than 10.1.6 because VMware Fusion has 10.1.5.4545 and it's unclear by the wording of the blog if it includes revisions or not).
There is an additional check you could do to see if netsh interface tcp show global shows the Receive-Side Scaling State set to enabled or disabled, but I didn't work out where that was stored. This should get you going…
But wait… there's more
If you saw my webinar on operational validation of SQL Server at scale you may already have your Jenkins instance running and be prepared to validate this daily across hundreds or thousands of servers in literally less than a minute every day, and give you results on a dashboard.
But even if you haven't seen it yet, if you Install-Module Jojoba you can easily create a function which will automatically run your validation at scale right from the command line. How? Get-Help about_Jojoba to see the template in Example 1 and then slot the code in with a Write-JojobaFail where you want to fail the test.
function Test-VmwareToolsRss {
[CmdletBinding()]
param (
[Parameter(Mandatory=$true, ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)]
[Alias("ComputerName")]
[string] $InputObject,
[string] $JojobaBatch = [System.Guid]::NewGuid().ToString(),
[int] $JojobaThrottle = $env:NUMBER_OF_PROCESSORS
)
begin {
}
process {
Start-Jojoba {
$cimSession = New-CimSessionDown $InputObject
$networkAdapters = Get-CimInstance -OperationTimeoutSec 30 -CimSession $cimSession -Class Win32_NetworkAdapter -Filter "Manufacturer = 'VMware, Inc.' Or Manufacturer = 'Intel Corporation'"
foreach ($networkAdapter in $networkAdapters) {
$setting = Get-CimRegValue -CimSession $cimSession -Key "SYSTEM\CurrentControlSet\Control\Class\{4d36e972-e325-11ce-bfc1-08002be10318}\$($networkAdapter.DeviceID.PadLeft(4, '0'))" -Value "*RSS" | Select-Object -ExpandProperty Data
if ($setting -eq 1) {
$imagePath = Get-CimRegValue -CimSession $cimSession -Key "SYSTEM\CurrentcontrolSet\Services\VMTools" -Value ImagePath | Select-Object -ExpandProperty Data
if ($imagePath -match '"(.*)"') {
$imagePath = $Matches[1]
}
$imagePath = "\\$InputObject\$($imagePath.Replace(':', '$'))"
$image = Get-Item $imagePath
if ($image.VersionInfo.FileVersionRaw -ge [version] "9.10" -and $image.VersionInfo.FileVersionRaw -lt [version] "10.1.6") {
Write-JojobaFail "Network adapter $($networkAdapter.Name) is enabled and on VMware tools $($image.VersionInfo.FileVersion)"
}
}
}
}
}
end {
Publish-Jojoba
}
}
The end result?
And so on. VMware Fusion's Intel drivers aren't explicitly called out in the blog post as having this issue but it's a great demo of seeing a problem and building a test on the spot to quickly validate your entire server fleet! And it's useful and works, too. You should probably run this right now.