What would you do with this crash?

Michael_Natkin · February 3, 2009, 6:36pm

I’ve got a NULL_POINTER_READ in this minidump with virtually nothing in the stack except what appears to be a completely bogus address (note big offset). I’ve got my symbols and image set correctly. I don’t have a way to repro this crash, though we have received quite a few dumps that look like this. I’ve tried a “dps esp-100 esp+100” and don’t see any useful symbols there. What would be my next steps in trying to find a real call stack? Any thoughts appreciated!

FAULTING_IP:
AfterFX_10000000!controlfp_s+1585f6
1015c65a 8b10 mov edx,dword ptr [eax]

EXCEPTION_RECORD: ffffffff – (.exr 0xffffffffffffffff)
ExceptionAddress: 1015c65a (AfterFX_10000000!controlfp_s+0x001585f6)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000000
Parameter[1]: 00000000
Attempt to read from address 00000000

DEFAULT_BUCKET_ID: NULL_POINTER_READ

PROCESS_NAME: AfterFX.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_PARAMETER1: 00000000

EXCEPTION_PARAMETER2: 00000000

READ_ADDRESS: 00000000

FOLLOWUP_IP:
AfterFX_10000000!controlfp_s+1585f6
1015c65a 8b10 mov edx,dword ptr [eax]

FAULTING_THREAD: 0000054c

PRIMARY_PROBLEM_CLASS: NULL_POINTER_READ

BUGCHECK_STR: APPLICATION_FAULT_NULL_POINTER_READ

LAST_CONTROL_TRANSFER: from 00000000 to 1015c65a

STACK_TEXT:
008fc4b0 00000000 008fd98c 6cee13c3 008fd734 AfterFX_10000000!controlfp_s+0x1585f6

STACK_COMMAND: ~0s; .ecxr ; kb

SYMBOL_STACK_INDEX: 0

SYMBOL_NAME: AfterFX_10000000!controlfp_s+1585f6

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: AfterFX_10000000

IMAGE_NAME: AfterFX.dll

DEBUG_FLR_IMAGE_TIMESTAMP: 48be54e6

FAILURE_BUCKET_ID: NULL_POINTER_READ_c0000005_AfterFX.dll!controlfp_s

BUCKET_ID: APPLICATION_FAULT_NULL_POINTER_READ_AfterFX_10000000!controlfp_s+1585f6

Tim_Roberts · February 3, 2009, 6:51pm

xxxxx@adobe.com wrote:

I’ve got a NULL_POINTER_READ in this minidump with virtually nothing in the stack except what appears to be a completely bogus address (note big offset). I’ve got my symbols and image set correctly. I don’t have a way to repro this crash, though we have received quite a few dumps that look like this. I’ve tried a “dps esp-100 esp+100” and don’t see any useful symbols there. What would be my next steps in trying to find a real call stack? Any thoughts appreciated!

FAULTING_IP:
AfterFX_10000000!controlfp_s+1585f6
1015c65a 8b10 mov edx,dword ptr [eax]

It looks like it doesn’t really have symbols for this particular version
of AfterFX.exe. In that case, the best you can do is go back to the
link map. You do grab a map every time you build, don’t you?

It’s unusual to have an exe with a preferred load address of 10000000.
Something you override on purpose?

EXCEPTION_RECORD: ffffffff – (.exr 0xffffffffffffffff)
ExceptionAddress: 1015c65a (AfterFX_10000000!controlfp_s+0x001585f6)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000000
Parameter[1]: 00000000
Attempt to read from address 00000000

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

jimd · February 3, 2009, 7:04pm

What you need to do is build the pdb for the release version, in which case
it will go right to the line of code in the source (assuming that you have
the symbol paths set right, and the original source trees in the right
place).
If you can recreate the source tree for that version, then you can still
build it.

This is easier than rooting thru the map (and if you like the map file, it
can be generated from the PDB).

If you crash the stack bad enough, then the mini dump will be trashed too.

On Tue, Feb 3, 2009 at 6:50 PM, Tim Roberts wrote:

> xxxxx@adobe.com wrote:
> > I’ve got a NULL_POINTER_READ in this minidump with virtually nothing in
> the stack except what appears to be a completely bogus address (note big
> offset). I’ve got my symbols and image set correctly. I don’t have a way to
> repro this crash, though we have received quite a few dumps that look like
> this. I’ve tried a “dps esp-100 esp+100” and don’t see any useful symbols
> there. What would be my next steps in trying to find a real call stack? Any
> thoughts appreciated!
> >
> > FAULTING_IP:
> > AfterFX_10000000!controlfp_s+1585f6
> > 1015c65a 8b10 mov edx,dword ptr [eax]
> >
>
> It looks like it doesn’t really have symbols for this particular version
> of AfterFX.exe. In that case, the best you can do is go back to the
> link map. You do grab a map every time you build, don’t you?
>
> It’s unusual to have an exe with a preferred load address of 10000000.
> Something you override on purpose?
>
> > EXCEPTION_RECORD: ffffffff – (.exr 0xffffffffffffffff)
> > ExceptionAddress: 1015c65a (AfterFX_10000000!controlfp_s+0x001585f6)
> > ExceptionCode: c0000005 (Access violation)
> > ExceptionFlags: 00000000
> > NumberParameters: 2
> > Parameter[0]: 00000000
> > Parameter[1]: 00000000
> > Attempt to read from address 00000000
> >
>
> –
> Tim Roberts, xxxxx@probo.com
> Providenza & Boekelheide, Inc.
>
>
> —
> WINDBG is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

Michael_Natkin · February 3, 2009, 7:54pm

You know, I think you guys are on to something. It isn’t that we don’t have the .pdb exactly. We stupidly have two modules with the same base name: AfterFX.dll and AfterFX.exe. The .exe is just a tiny shell, the code we really care about is in the .dll. But I think the debugging tools are confused by this. Anyone have an idea how to tell it to ignore the .exe and manually use the symbols from the .dll and the corresponding .pdb? Clearly we need to change this bit of architecture going forward!

Ken_Johnson · February 3, 2009, 11:25pm

They should normally figure it out if you use the symbol server or the codeview record burned into the image &&

? S

-----Original Message-----
From: xxxxx@adobe.com
Sent: Tuesday, February 03, 2009 16:54
To: Kernel Debugging Interest List
Subject: RE:[windbg] What would you do with this crash?

You know, I think you guys are on to something. It isn’t that we don’t have the .pdb exactly. We stupidly have two modules with the same base name: AfterFX.dll and AfterFX.exe. The .exe is just a tiny shell, the code we really care about is in the .dll. But I think the debugging tools are confused by this. Anyone have an idea how to tell it to ignore the .exe and manually use the symbols from the .dll and the corresponding .pdb? Clearly we need to change this bit of architecture going forward!

—
WINDBG is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Ken_Johnson · February 3, 2009, 11:31pm

Let’s try that again.

They should normally figure it out if you use the symbol server or the codeview record burned into the image && pointing to the pdb on the build machine.

An example of this working all the time is wow64 debugging with the 64-bit debugger, where there are two ntdll images. Matching is done for binaries looked up via sympath based on image size + timestamp, and for various methods (currently a guid + PdbAge, IIRC) for pdbs looked up via sympathy.

Can you use !sym noisy and reload symbols and see what the output is?

S

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Skywing
Sent: Tuesday, February 03, 2009 11:25 PM
To: Kernel Debugging Interest List
Subject: RE: RE:[windbg] What would you do with this crash?

They should normally figure it out if you use the symbol server or the codeview record burned into the image &&

? S

-----Original Message-----
From: xxxxx@adobe.com
Sent: Tuesday, February 03, 2009 16:54
To: Kernel Debugging Interest List
Subject: RE:[windbg] What would you do with this crash?

You know, I think you guys are on to something. It isn’t that we don’t have the .pdb exactly. We stupidly have two modules with the same base name: AfterFX.dll and AfterFX.exe. The .exe is just a tiny shell, the code we really care about is in the .dll. But I think the debugging tools are confused by this. Anyone have an idea how to tell it to ignore the .exe and manually use the symbols from the .dll and the corresponding .pdb? Clearly we need to change this bit of architecture going forward!

—
WINDBG is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

—
WINDBG is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Michael_Natkin · February 4, 2009, 11:44am

Thanks for the advice! Below is the output of !sym noisy; .reload /i for AfterFX.exe and AfterFX.pdb. It is normal that the .pdbs are thought to mismatch, as we have had to rebuild them since the original build machine, but they are from the exact same changelist, so we know they are in fact valid. But if I do “x AfterFX!” I see only the symbols from AfterFX.exe and none of the ones from AfterFX.pdb, and they never show up in my stack traces though they should be at the bottom of every one. Any thoughts on how to get them recognized? Thanks a million! Oh, and “Z:\ae_eng\symbolstore\ae90\AfterFX.pdb” is in fact the symbols for the .dll, not the .exe.

DBGHELP: C:\Program Files (x86)\Adobe\Adobe After Effects CS4\Support Files\AfterFX.exe - OK
DBGENG: C:\Program Files (x86)\Adobe\Adobe After Effects CS4\Support Files\AfterFX.exe - Mapped image memory
DBGHELP: Z:\ae_eng\symbolstore\ae90\AfterFX.pdb - mismatched pdb
DBGHELP: Z:\ae_eng\symbolstore\ae90\exe\AfterFX.pdb - file not found
DBGHELP: Z:\ae_eng\symbolstore\ae90\symbols\exe\AfterFX.pdb - file not found
SYMSRV: C:\Program Files\Debugging Tools for Windows (x64)\sym\AfterFX.pdb\35D331EF7FBA4B31BF756B52FE4BF2ED3\AfterFX.pdb not found
SYMSRV: http://msdl.microsoft.com/download/symbols/AfterFX.pdb/35D331EF7FBA4B31BF756B52FE4BF2ED3/AfterFX.pdb not found
DBGHELP: AfterFX.pdb - file not found
DBGHELP: c:\Chinchillada\pro\lib\win32\rel\AfterFX.pdb - file not found
DBGHELP: Loaded mismatched pdb for AfterFX.exe
DBGENG: AfterFX.exe has mismatched symbols - type “.hh dbgerr003” for details
DBGHELP: AfterFX - private symbols & lines
Z:\ae_eng\symbolstore\ae90\AfterFX.pdb - unmatched

DBGHELP: C:\Program Files (x86)\Adobe\Adobe After Effects CS4\Support Files\AfterFX.dll - OK
DBGENG: C:\Program Files (x86)\Adobe\Adobe After Effects CS4\Support Files\AfterFX.dll - Mapped image memory
DBGHELP: Z:\ae_eng\symbolstore\ae90\AfterFX.pdb - mismatched pdb
DBGHELP: Z:\ae_eng\symbolstore\ae90\dll\AfterFX.pdb - file not found
DBGHELP: Z:\ae_eng\symbolstore\ae90\symbols\dll\AfterFX.pdb - file not found
SYMSRV: C:\Program Files\Debugging Tools for Windows (x64)\sym\AfterFX.pdb\35D331EF7FBA4B31BF756B52FE4BF2ED4\AfterFX.pdb not found
SYMSRV: http://msdl.microsoft.com/download/symbols/AfterFX.pdb/35D331EF7FBA4B31BF756B52FE4BF2ED4/AfterFX.pdb not found
DBGHELP: AfterFX.pdb - file not found
DBGHELP: c:\Chinchillada\pro\lib\win32\rel\AfterFX.pdb - file not found
DBGHELP: Loaded mismatched pdb for AfterFX.dll
DBGENG: AfterFX.dll has mismatched symbols - type “.hh dbgerr003” for details
DBGHELP: AfterFX_10000000 - private symbols & lines
Z:\ae_eng\symbolstore\ae90\AfterFX.pdb - unmatched

Michael_Natkin · February 4, 2009, 11:47am

One more data point. Following the notes on this page:

http://www.wintellect.com/cs/blogs/jrobbins/archive/2007/10/28/itoldyouso-an-undocumented-windbg-extension-command.aspx

I got the following. Note interesting last line.

AfterFX.exe
Timestamp: 48BE3BC7
SizeOfImage: F9000
pdb: c:\Chinchillada\pro\lib\win32\rel\AfterFX.pdb
pdb sig: 35D331EF-7FBA-4B31-BF75-6B52FE4BF2ED
age: 3

Loaded pdb is Z:\ae_eng\symbolstore\ae90\AfterFX.pdb

AfterFX.pdb
pdb sig: BDCFEBB0-DE57-45EF-94C0-8DF47D444F37
age: 3

sig MISMATCH: AfterFX.pdb and AfterFX.exe

jimd · February 4, 2009, 12:06pm

Will, you should NOT have a dll and exe named the same.
Since you have 2 pdbs with the same name, it could very well confuse windbg.

On Wed, Feb 4, 2009 at 11:47 AM, wrote:

> One more data point. Following the notes on this page:
>
>
> http://www.wintellect.com/cs/blogs/jrobbins/archive/2007/10/28/itoldyouso-an-undocumented-windbg-extension-command.aspx
>
> I got the following. Note interesting last line.
>
>
> AfterFX.exe
> Timestamp: 48BE3BC7
> SizeOfImage: F9000
> pdb: c:\Chinchillada\pro\lib\win32\rel\AfterFX.pdb
> pdb sig: 35D331EF-7FBA-4B31-BF75-6B52FE4BF2ED
> age: 3
>
> Loaded pdb is Z:\ae_eng\symbolstore\ae90\AfterFX.pdb
>
> AfterFX.pdb
> pdb sig: BDCFEBB0-DE57-45EF-94C0-8DF47D444F37
> age: 3
>
> sig MISMATCH: AfterFX.pdb and AfterFX.exe
>
> —
> WINDBG is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

Michael_Natkin · February 4, 2009, 12:27pm

Yes, having the same name is terrible. Unfortunately, we already shipped this way and we have stacks we really want to analyze, so we are trying to make do. The next version of the app won’t have this problem, that is for sure!

I did just discover a somewhat useable workaround. First I do “.reload /i” to load everything inlcluding all of those .pdbs whose dates are wrong because we don’t have the real ones from the build machine. The I do “.reload /f AfterFX.dll”, which will cause it to ignore all .pdbs and use the dll export symbols for just that one module. That seems to be close enough for a lot of the cases I’m interested in.

Ken_Johnson · February 4, 2009, 1:33pm

Is Z:\ae_eng\symbolstore a real symbol server repository, or just a directory where symbols have been manually copied to?

If you create a symbol server using symstore.exe, it will lay things out such that the debugger doesn’t need to rely on filenames alone to retrieve symbol data (instead, as previously mentioned, binary size + link timestamp + filename, and pdb guid + pdb age + filename are used for binaries and new-style pdbs, restprctively; for VC6-era pdbs, there’s some other unique differentiator burned into them, though doesn’t look like a guid to me).

A symbol server repository can thus also contain multiple versions of the same file, by the same mechanism.

S

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@adobe.com
Sent: Wednesday, February 04, 2009 11:44 AM
To: Kernel Debugging Interest List
Subject: RE:[windbg] What would you do with this crash?

Thanks for the advice! Below is the output of !sym noisy; .reload /i for AfterFX.exe and AfterFX.pdb. It is normal that the .pdbs are thought to mismatch, as we have had to rebuild them since the original build machine, but they are from the exact same changelist, so we know they are in fact valid. But if I do “x AfterFX!” I see only the symbols from AfterFX.exe and none of the ones from AfterFX.pdb, and they never show up in my stack traces though they should be at the bottom of every one. Any thoughts on how to get them recognized? Thanks a million! Oh, and “Z:\ae_eng\symbolstore\ae90\AfterFX.pdb” is in fact the symbols for the .dll, not the .exe.

DBGHELP: C:\Program Files (x86)\Adobe\Adobe After Effects CS4\Support Files\AfterFX.exe - OK
DBGENG: C:\Program Files (x86)\Adobe\Adobe After Effects CS4\Support Files\AfterFX.exe - Mapped image memory
DBGHELP: Z:\ae_eng\symbolstore\ae90\AfterFX.pdb - mismatched pdb
DBGHELP: Z:\ae_eng\symbolstore\ae90\exe\AfterFX.pdb - file not found
DBGHELP: Z:\ae_eng\symbolstore\ae90\symbols\exe\AfterFX.pdb - file not found
SYMSRV: C:\Program Files\Debugging Tools for Windows (x64)\sym\AfterFX.pdb\35D331EF7FBA4B31BF756B52FE4BF2ED3\AfterFX.pdb not found
SYMSRV: http://msdl.microsoft.com/download/symbols/AfterFX.pdb/35D331EF7FBA4B31BF756B52FE4BF2ED3/AfterFX.pdb not found
DBGHELP: AfterFX.pdb - file not found
DBGHELP: c:\Chinchillada\pro\lib\win32\rel\AfterFX.pdb - file not found
DBGHELP: Loaded mismatched pdb for AfterFX.exe
DBGENG: AfterFX.exe has mismatched symbols - type “.hh dbgerr003” for details
DBGHELP: AfterFX - private symbols & lines
Z:\ae_eng\symbolstore\ae90\AfterFX.pdb - unmatched

DBGHELP: C:\Program Files (x86)\Adobe\Adobe After Effects CS4\Support Files\AfterFX.dll - OK
DBGENG: C:\Program Files (x86)\Adobe\Adobe After Effects CS4\Support Files\AfterFX.dll - Mapped image memory
DBGHELP: Z:\ae_eng\symbolstore\ae90\AfterFX.pdb - mismatched pdb
DBGHELP: Z:\ae_eng\symbolstore\ae90\dll\AfterFX.pdb - file not found
DBGHELP: Z:\ae_eng\symbolstore\ae90\symbols\dll\AfterFX.pdb - file not found
SYMSRV: C:\Program Files\Debugging Tools for Windows (x64)\sym\AfterFX.pdb\35D331EF7FBA4B31BF756B52FE4BF2ED4\AfterFX.pdb not found
SYMSRV: http://msdl.microsoft.com/download/symbols/AfterFX.pdb/35D331EF7FBA4B31BF756B52FE4BF2ED4/AfterFX.pdb not found
DBGHELP: AfterFX.pdb - file not found
DBGHELP: c:\Chinchillada\pro\lib\win32\rel\AfterFX.pdb - file not found
DBGHELP: Loaded mismatched pdb for AfterFX.dll
DBGENG: AfterFX.dll has mismatched symbols - type “.hh dbgerr003” for details
DBGHELP: AfterFX_10000000 - private symbols & lines
Z:\ae_eng\symbolstore\ae90\AfterFX.pdb - unmatched

WINDBG is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer