MS symbol server is slow or broken again.

NTDEV
1

Hi!

It seems that a problems with the MS symbol server are coming back:

https://social.msdn.microsoft.com/Forums/en-US/35a31403-24f5-4552-8e3c-4c65f2b98c17/microsoft-symbol-server-is-slowbroken?forum=windbg

I have same troubles.

kd> .reload
SYMSRV: File: ntkrpamp.pdb

SYMSRV: Notifies the client application that a proxy has been detected.
SYMSRV: Connecting to the Server: http://msdl.microsoft.com/download/symbols.
SYMSRV: Successfully connected to the Server.
SYMSRV: Sending the information request to the server.
SYMSRV: Successfully sent the information request to the server.
SYMSRV: Waiting for the server to respond to a request.

SYMSRV: Closing the connection to the Server.
SYMSRV: Successfully closed the connection to the Server.
SYMSRV: ??? ??? ?? ???
SYMSRV: Get File Path: /download/symbols/ntkrpamp.pdb/684DA42A30CC450F81C535B4D18944B12/ntkrpamp.pdb

SYMSRV: Notifies the client application that a proxy has been detected.
SYMSRV: Connecting to the Server: http://msdl.microsoft.com/download/symbols.
SYMSRV: Successfully connected to the Server.
SYMSRV: Sending the information request to the server.
SYMSRV: Successfully sent the information request to the server.
SYMSRV: Waiting for the server to respond to a request.

SYMSRV: Closing the connection to the Server.
SYMSRV: Successfully closed the connection to the Server.
SYMSRV: ??? ??? ?? ???
SYMSRV: c:\symbols\ntkrpamp.pdb\684DA42A30CC450F81C535B4D18944B12\ntkrpamp.pdb not found
SYMSRV: http://msdl.microsoft.com/download/symbols/ntkrpamp.pdb/684DA42A30CC450F81C535B4D18944B12/ntkrpamp.pdb not found
DBGHELP: ntkrpamp.pdb - file not found
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ntkrpamp.exe -
DBGHELP: nt - export symbols
Loading Kernel Symbols
…
…
…
Loading User Symbols
…
…
…
…
…
Loading unloaded module list
…
SYMSRV: File: ntdll.dll

SYMSRV: Get File Path: /download/symbols/ntdll.dll/4CE7B96E13c000/ntdll.dll

SYMSRV: Notifies the client application that a proxy has been detected.
SYMSRV: Connecting to the Server: http://msdl.microsoft.com/download/symbols.
SYMSRV: Successfully connected to the Server.
SYMSRV: Sending the information request to the server.
SYMSRV: Successfully sent the information request to the server.
SYMSRV: Waiting for the server to respond to a request.
SYMSRV: Closing the connection to the Server.
SYMSRV: Successfully closed the connection to the Server.
SYMSRV: ??? ??? ?? ???
SYMSRV: c:\symbols\ntdll.dll\4CE7B96E13c000\ntdll.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/ntdll.dll/4CE7B96E13c000/ntdll.dll not found
DBGHELP: C:\Program Files\Windows Kits\8.1\Debuggers\ntdll.dll - file not found
DBGHELP: C:\Program Files\Windows Kits\8.1\Debuggers\ntdll.dll - file not found
DBGENG: C:\Windows\SYSTEM32\ntdll.dll image header does not match memory image header.
DBGENG: C:\Windows\SYSTEM32\ntdll.dll - Couldn’t map image from disk.
DBGHELP: No debug info for ntdll.dll. Searching for dbg file
SYMSRV: File: ntdll.dbg

SYMSRV: Get File Path: /download/symbols/ntdll.dbg/4CE7B96E13c000/ntdll.dbg

SYMSRV: Notifies the client application that a proxy has been detected.
SYMSRV: Connecting to the Server: http://msdl.microsoft.com/download/symbols.
SYMSRV: Successfully connected to the Server.
SYMSRV: Sending the information request to the server.
SYMSRV: Successfully sent the information request to the server.
SYMSRV: Waiting for the server to respond to a request.
SYMSRV: Closing the connection to the Server.
SYMSRV: Successfully closed the connection to the Server.
SYMSRV: ??? ??? ?? ???
SYMSRV: c:\symbols\ntdll.dbg\4CE7B96E13c000\ntdll.dbg not found
SYMSRV: http://msdl.microsoft.com/download/symbols/ntdll.dbg/4CE7B96E13c000/ntdll.dbg not found
DBGHELP: .\ntdll.dbg - file not found
DBGHELP: .\dll\ntdll.dbg - path not found
DBGHELP: .\symbols\dll\ntdll.dbg - path not found
DBGHELP: ntdll.dll missing debug info. Searching for pdb anyway
DBGHELP: Can’t use symbol server for ntdll.pdb - no header information available
DBGHELP: ntdll.pdb - file not found
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll -
DBGHELP: ntdll - export symbols

SYMSRV: File: i8042prt.pdb

SYMSRV: Get File Path: /download/symbols/i8042prt.pdb/1A67186BD81F424E89EB08442D81FF042/i8042prt.pdb

SYMSRV: Notifies the client application that a proxy has been detected.
SYMSRV: Connecting to the Server: http://msdl.microsoft.com/download/symbols.
SYMSRV: Successfully connected to the Server.
SYMSRV: Sending the information request to the server.
SYMSRV: Successfully sent the information request to the server.
SYMSRV: Waiting for the server to respond to a request.

SYMSRV: Closing the connection to the Server.
SYMSRV: Successfully closed the connection to the Server.
SYMSRV: ??? ??? ?? ???
SYMSRV: c:\symbols\i8042prt.pdb\1A67186BD81F424E89EB08442D81FF042\i8042prt.pdb not found
SYMSRV: http://msdl.microsoft.com/download/symbols/i8042prt.pdb/1A67186BD81F424E89EB08442D81FF042/i8042prt.pdb not found
DBGHELP: i8042prt.pdb - file not found
*** ERROR: Module load completed but symbols could not be loaded for i8042prt.sys
DBGHELP: i8042prt - no symbols loaded

************* Symbol Loading Error Summary **************
Module name Error
ntkrpamp The system cannot find the file specified
The SYMSRV client failed to find a file in the UNC store, or there
is an invalid UNC store (an invalid path or the pingme.txt file is
not present in the root directory), or the file is present in the
symbol server exclusion list.

i8042prt The operation timed out : srv*c:\symbols*http://msdl.microsoft.com/download/symbols
The SYMSRV client failed connecting to the symbol server.

ntdll No data is available : srv*c:\symbols*http://msdl.microsoft.com/download/symbols
The SYMSRV client failed validating the GUID for the module, or the
module does not have the debug header information.

The local symbol cache directory is still empty.
Sometimes it working and some symbols are loading, but very-very slow.

2

Yes. This is a well known issue. We’ve been reporting it to the community over the past couple of months (https://twitter.com/osrdrivers).

As we posted before, MSFT is aware of the problem and is actively working on it. It’s not something simple that they can easily fix.

The best work-around is to download the symbols you need and then stop using the MSFT symbol server (by using the locally cached symbols or by setting up your own internal symbol server).

Yes, it’s annoying. Here at OSR we’re suffering to. Ugh.

Peter
OSR
@OSRDrivers

3

> MSFT is aware of the problem and is actively working on it.

Thank you, Peter.
This is a good news for me.

4

The recipe is:
Every so often, move whatever directories were created in the srv symbol store directory to your static symbol directory. Make sure the static directory is before the symbol server in the symbol path.

I usually work without having MS server in the sym path, and only add it temporarily when it needs new symbols after OS was updated.

5

I never appreciated how heavily I depended on the symbol server until it started having issues. I’m constantly switching targets and needing new symbols, removing the server from my path is a major annoyance and reduces my productivity (though leaving it in and dealing with insane timeouts also reduces my productivity).

-scott
OSR
@OSRDrivers