No solutions today, only sadness and finger pointing.

Part 1

If you're using PowerShell and the new SQL Server SQLServer PSProvider that gets installed in the newer version or otherwise comes free with one of the feature packs, you are probably dying to try out the new object-based SQL Server file system. Until you actually try to use it:

Import-Module SQLPS -DisableNameChecking

New-PSDrive -Name MYSERVER -PSProvider SQLServer -Root SQLSERVER:\SQL\MYSERVER\DEFAULT -Credential MySQLUserName
WARNING: Could not obtain SQL Server Service information. An attempt to connect to WMI on 'MYSERVER' failed with the following error: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
WARNING: Could not obtain SQL Server Service information. An attempt to connect to WMI on 'MYSERVER' failed with the following error: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
WARNING: Could not obtain SQL Server Service information. An attempt to connect to WMI on 'MYSERVER' failed with the following error: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
WARNING: Could not obtain SQL Server Service information. An attempt to connect to WMI on 'MYSERVER' failed with the following error: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
New-PSDrive : SQL Server PowerShell provider error: Could not connect to 'MYSERVERDEFAULT'. [Object reference not set to an instance of an object.]
At line:1 char:1
+ New-PSDrive -Name MyServer -PSProvider SQLServer -Root SQLSERVER:SQLMYSERVERD ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OpenError: (SQLSERVER:SQLMYSERVERDEFAULT:SqlPath) [New-PSDrive], GenericProviderException
    + FullyQualifiedErrorId : ConnectFailed,Microsoft.PowerShell.Commands.NewPSDriveCommand

Oops!

That's because the Credential parameter, while technically allowing you to use the SQL Server authentication instead of domain authentication, still checks to make sure your domain credentials have access to WMI on the server you are connecting to.

Which makes it utterly useless for DBAs administering servers across multiple untrusteed domains.

What makes it even more infuriating is that in those situations, you likely have WMI access over other credentials. For example, this would work:

Get-WmiObject win32_Bios -ComputerName MYSERVER -Credential Administrator

This returns a few lines of BIOS information.

But because you can't tell the PSProvider to use those credentials, it will just fail. Bad Microsoft. Bad.

Part 2

And while investigating the above WMI testing, I found one server I could not connect to, where I would get this error message:

Get-WmiObject : The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)
At line:1 char:1
+ Get-WmiObject win32_bios -ComputerName MYSERVER -Credential Administrator
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (:) [Get-WmiObject], COMException
+ FullyQualifiedErrorId : GetWMICOMException,Microsoft.PowerShell.Commands.GetWmiObjectCommand

Which was weird because the RPC services were running, I could run the command locally on that server and it was fine, and I could telnet over WMI TCP Port 135 showing that port was opened.

What followed was some digging into Wireshark and comparing between a server where WMI works and a server where WMI doesn't work. This is the working version:

Source Destination Protocol Length Info
Local Working Remote TCP 66 49864 > epmap [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
Working Remote Local TCP 66 epmap > 49864 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1452 WS=256 SACK_PERM=1
Local Working Remote TCP 54 49864 > epmap [ACK] Seq=1 Ack=1 Win=66560 Len=0
Local Working Remote DCERPC 170 Bind: call_id: 2 Fragment: Single, 2 context items, 1st IOXIDResolver V0.0
Working Remote Local DCERPC 138 Bind_ack: call_id: 2 Fragment: Single Unknown result (3), reason: Local limit exceeded
Local Working Remote IOXIDResolver 78 ServerAlive2 request
Working Remote Local IOXIDResolver 290 ServerAlive2 response[Long frame (2 bytes)]
Local Working Remote TCP 66 49865 > epmap [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
Local Working Remote TCP 54 49864 > epmap [ACK] Seq=141 Ack=321 Win=66304 Len=0
Working Remote Local TCP 66 epmap > 49865 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1452 WS=256 SACK_PERM=1
Local Working Remote TCP 54 49865 > epmap [ACK] Seq=1 Ack=1 Win=66560 Len=0
Local Working Remote DCERPC 174 Bind: call_id: 3 Fragment: Single ISystemActivator V0.0, NTLMSSP_NEGOTIATE
Working Remote Local DCERPC 290 Bind_ack: call_id: 3 Fragment: Single, NTLMSSP_CHALLENGE accept max_xmit: 5840 max_recv: 5840
Local Working Remote DCERPC 532 AUTH3: call_id: 3 Fragment: Single, NTLMSSP_AUTH, User: REMOTEDOMAIN\REMOTEUSER
Local Working Remote ISystemActivator 870 RemoteCreateInstance request
Working Remote Local TCP 60 epmap > 49865 [ACK] Seq=237 Ack=1415 Win=65280 Len=0
Working Remote Local ISystemActivator 1238 RemoteCreateInstance response[Malformed Packet]
Local Working Remote TCP 66 49868 > 49154 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
Local Working Remote TCP 54 49865 > epmap [ACK] Seq=1415 Ack=1421 Win=65280 Len=0
Working Remote Local TCP 66 49154 > 49868 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1452 WS=256 SACK_PERM=1

And this is the non-working version:

Source Destination Protocol Length Info
Local Failing Remote TCP 66 49884 > epmap [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
Failing Remote Local TCP 66 epmap > 49884 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1452 WS=256 SACK_PERM=1
Local Failing Remote TCP 54 49884 > epmap [ACK] Seq=1 Ack=1 Win=66560 Len=0
Local Failing Remote DCERPC 170 Bind: call_id: 2 Fragment: Single, 2 context items, 1st IOXIDResolver V0.0
Failing Remote Local DCERPC 138 Bind_ack: call_id: 2 Fragment: Single Unknown result (3), reason: Local limit exceeded
Local Failing Remote IOXIDResolver 78 ServerAlive2 request
Failing Remote Local IOXIDResolver 266 ServerAlive2 response
Local Failing Remote TCP 66 49885 > epmap [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
Failing Remote Local TCP 66 epmap > 49885 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1452 WS=256 SACK_PERM=1
Local Failing Remote TCP 54 49885 > epmap [ACK] Seq=1 Ack=1 Win=66560 Len=0
Local Failing Remote TCP 54 49884 > epmap [ACK] Seq=141 Ack=297 Win=66304 Len=0
Local Failing Remote DCERPC 174 Bind: call_id: 3 Fragment: Single ISystemActivator V0.0, NTLMSSP_NEGOTIATE
Local Failing Remote DCERPC 174 [TCP Retransmission] Bind: call_id: 3 Fragment: Single ISystemActivator V0.0, NTLMSSP_NEGOTIATE
Failing Remote Local DCERPC 300 Bind_ack: call_id: 3 Fragment: Single, NTLMSSP_CHALLENGE accept max_xmit: 5840 max_recv: 5840
Local Failing Remote DCERPC 528 AUTH3: call_id: 3 Fragment: Single, NTLMSSP_AUTH, User: REMOTEDOMAIN\REMOTEUSER
Local Failing Remote ISystemActivator 870 RemoteCreateInstance request
Failing Remote Local TCP 66 [TCP Dup ACK 14#1] epmap > 49885 [ACK] Seq=247 Ack=121 Win=66560 Len=0 SLE=1 SRE=121
Failing Remote Local TCP 60 epmap > 49885 [ACK] Seq=247 Ack=1411 Win=65280 Len=0
Failing Remote Local ISystemActivator 1182 RemoteCreateInstance response[Malformed Packet]
Local Failing Remote TCP 66 49886 > 49154 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
Failing Remote Local TCP 60 49154 > 49886 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
Local Failing Remote TCP 54 49885 > epmap [ACK] Seq=1411 Ack=1375 Win=65280 Len=0

The difference was apparent about a page down where trying to connect over another WMI TCP Port 49154. That was running on the server, but I could not connect to it. Just to be sure, I checked the Advanced Firewall settings on that server and did not find anything, and for a last check, ran this from a command prompt to make sure the port was open:

netstat -a -n

And then started up a session that will write a text log that should show if the Windows firewall has dropped a packet:

cd %userprofile%\desktop
netsh wfp capture start
...
netsh wfp capture stop

And it hadn't. This pretty much confirmed there as a router configuration blocking access along the way.