wow64exts, or, where are the 32 bit stacks

Hoping for a bit of guidance here from folk with more experience of using wow64exts.

OK. So. Driver developer here: when I fire up WinDbg it’s normally to do some kernel-mode debugging. However, sometimes when people’s machines lock up, they either generate a full memory dump or we attach WinDbg and pull one down the wire.
Often the cause of the lockup lies in the (32 bit) user mode code, so ideally we’d like to analyze these dumps and see the 32 bit stacks even if the dump came from a 64 bit box. But I can’t seem to get 'em.

I’ve been given one such dump to look at to try and *find* the 32 bit stacks.
Well I can do a !process 0 0 and see the 32 bit process there - fine. I can dump its thread stacks (lots of wow64cpu!blah at the bottom… okay…).

0: kd> .process fffffa80079e4b30
Implicit process is now fffffa80079e4b30 0: kd\> !wow64exts.sw Switched to 32bit mode 0: kd:x86\> k ^ Current scope machine type mismatch error in 'k' 0: kd:x86\> !wow64exts.k The context is partially valid. Only x86 user-mode context is available. Walking 64bit Stack... \*\*\* Stack trace for last set context - .thread/.cxr resets it Child-SP RetAddr Call Site fffff88007740710 fffff80002ad5752 nt!KiSwapContext+0x7a fffff88007740850 fffff80002ad78af nt!KiCommitThreadWait+0x1d2 fffff880077408e0 fffff80002aacfc8 nt!KeWaitForSingleObject+0x19f fffff88007740980 fffff80002aab009 nt!KiSuspendThread+0x74 fffff880077409c0 fffff80002ad595d nt!KiDeliverApc+0x211 fffff88007740a40 fffff80002ad78af nt!KiCommitThreadWait+0x3dd fffff88007740ad0 fffff80002dc9db2 nt!KeWaitForSingleObject+0x19f fffff88007740b70 fffff80002acd853 nt!NtWaitForSingleObject+0xb2 fffff88007740be0 0000000074cb2dd9 nt!KiSystemServiceCopyEnd+0x13 000000000008ebf8 0000000074cb2bcd 0x74cb2dd9 000000000008ec00 0000000077220190 0x74cb2bcd 000000000008ec08 0000000074d20023 0x77220190 000000000008ec10 00000000002cc5a8 0x74d20023 000000000008ec18 000000000018fff0 0x2cc5a8 000000000008ec20 000000000008ec50 0x18fff0 000000000008ec28 0000000000390008 0x8ec50 000000000008ec30 00000000`00000000 0x390008
Walking 32bit Stack…
^ Current scope machine type mismatch error in 'k ’
!wow64exts.k: Execute failed: 0x80040205

?? Current scope machine type mismatch error in 'k ’ ??

I suspect there is some key fact I’m not aware of here.

I did have a look in the windbg help but as I interpreted it, if you’re grabbing a *full* memory dump it doesn’t matter whether the machine running windbg - or the version of windbg being run - is 64 bit or not. So it must be something else - any help gratefully received.

OK, there’s certainly something wrong here. I’m not sure if this is a bug in
the current implementation or a limitation. I’ll demonstrate what I’m seeing
on a live dump as it ends better than the crash dump version :slight_smile:

I force a context switch to a 32bit process in WinDBG via .process /i, in
this case I picked 32bit IE:

0: kd> !process -1 0
PROCESS fffffa8002455340
SessionId: 1 Cid: 0aa0 Peb: 7efdf000 ParentCid: 0444
DirBase: 1a2a3000 ObjectTable: fffff8a002e36e40 HandleCount: 327.
Image: iexplore.exe

I then switch to a thread in the process and attempt to do !wow64exts.k,
which meets the same fate as yours:

0: kd> !wow64exts.k
Walking 64bit Stack…
*** Stack trace for last set context - .thread/.cxr resets it
Child-SP RetAddr Call Site
fffff8800473a660 fffff800026c2052 nt!KiSwapContext+0x7a
fffff8800473a7a0 fffff800026c41af nt!KiCommitThreadWait+0x1d2
fffff8800473a830 fffff800026d94ef nt!KeWaitForSingleObject+0x19f
fffff8800473a8d0 fffff800029be736 nt!AlpcpSignalAndWait+0x8f
fffff8800473a980 fffff800029bc6c0 nt!AlpcpReceiveSynchronousReply+0x46
fffff8800473a9e0 fffff800029b9c3b nt!AlpcpProcessSynchronousRequest+0x33d
fffff8800473ab00 fffff800026ba153 nt!NtAlpcSendWaitReceivePort+0x1ab
fffff8800473abb0 0000000077b6070a nt!KiSystemServiceCopyEnd+0x13
000000000371e758 000000007581e2f0 ntdll!ZwAlpcSendWaitReceivePort+0xa
000000000371e760 000000007580cf87 wow64!whNtAlpcSendWaitReceivePort+0x64
000000000371e7c0 000000007579276d wow64!Wow64SystemServiceEx+0xd7
000000000371f080 000000007580d07e
wow64cpu!TurboDispatchJumpAddressEnd+0x24
000000000371f140 000000007580c549 wow64!RunCpuSimulation+0xa
000000000371f190 0000000077b8d177 wow64!Wow64LdrpInitialize+0x429
000000000371f6e0 0000000077b4308e ntdll! ?? ::FNODOBFM::string'+0x2bfe4 000000000371f750 00000000`00000000 ntdll!LdrInitializeThunk+0xe
Walking 32bit Stack…
^ Current scope machine type mismatch error in 'k ’
!wow64exts.k: Execute failed: 0x80040205

If you hook a debugger up to this, the stack walk is failing by raising an
exception:

0:003> kc
Call Site
KERNELBASE!RaiseException
dbgeng!ReportError
dbgeng!WrapParseStackCmd
dbgeng!ProcessCommands
dbgeng!ProcessCommandsAndCatch
dbgeng!Execute
dbgeng!DebugClient::ExecuteWide
dbgeng!DebugClient::Execute
wow64exts!k

A call to GetExplicitContext fails, so DbgEng is reporting error 0x1038 (not
sure what that stands for).

After playing with it a bit more, I had a theory that the problem has to do
with the context at the time of the last debugger event. I think that
because we’re in 64bit code the 32bit stack walk command is getting confused
and refusing to run. So, I set a breakpoint in the wow64 layer, hit Go,
tried again, and the command works like a charm:

0: kd> ba e1 wow64!RunCpuSimulation
0: kd> g
Breakpoint 7 hit
wow64!RunCpuSimulation:
0033:000000007580d074 4883ec48 sub rsp,48h 1: kd\> .reload /user Loading User Symbols ..... Loading Wow64 Symbols ................................................................ ........ 1: kd\> !wow64exts.k Walking 64bit Stack... Child-SP RetAddr Call Site 000000000119d778 0000000075808a40 wow64!RunCpuSimulation 000000000119d780 00000000757d29ee wow64!Wow64KiUserCallbackDispatcher+0x204 000000000119dad0 0000000077b5fdf5 wow64win!whcbfnDWORD+0xe2 000000000119e4c0 00000000757dfbda ntdll!KiUserCallbackDispatcherContinue 000000000119e548 00000000757bab53 wow64win!NtUserPeekMessage+0xa 000000000119e550 000000007580cf87 wow64win!whNtUserPeekMessage+0x37 000000000119e5c0 000000007579276d wow64!Wow64SystemServiceEx+0xd7 000000000119ee80 000000007580d07e wow64cpu!TurboDispatchJumpAddressEnd+0x24 000000000119ef40 000000007580c549 wow64!RunCpuSimulation+0xa 000000000119ef90 0000000077b8d177 wow64!Wow64LdrpInitialize+0x429 000000000119f4e0 0000000077b4308e ntdll! ?? ::FNODOBFM::string’+0x2bfe4
000000000119f550 0000000000000000 ntdll!LdrInitializeThunk+0xe
Walking 32bit Stack…
ChildEBP RetAddr
02dcfb08 729f03d2 ntdll_77cf0000!KiUserCallbackDispatcher
02dcfb90 72a00446 IEFRAME!CTabWindow::_TabWindowThreadProc+0x24e
02dcfc48 761849bd IEFRAME!LCIETab_ThreadProc+0x2c1
02dcfc58 76673677 iertutil!CIsoScope::RegisterThread+0xab
02dcfc64 77d29d72 kernel32!BaseThreadInitThunk+0xe
02dcfca4 77d29d45 ntdll_77cf0000!__RtlUserThreadStart+0x70
02dcfcbc 00000000 ntdll_77cf0000!_RtlUserThreadStart+0x1b

This doesn’t bode overly well for doing this from a crash dump though I
don’t play with this all that often, so maybe there’s a workaround (or I’m
just insane and doing something wrong also).

-scott


Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

wrote in message news:xxxxx@windbg…
> Hoping for a bit of guidance here from folk with more experience of using
> wow64exts.
>
> OK. So. Driver developer here: when I fire up WinDbg it’s normally to do
> some kernel-mode debugging. However, sometimes when people’s machines lock
> up, they either generate a full memory dump or we attach WinDbg and pull
> one down the wire.
> Often the cause of the lockup lies in the (32 bit) user mode code, so
> ideally we’d like to analyze these dumps and see the 32 bit stacks even if
> the dump came from a 64 bit box. But I can’t seem to get 'em.
>
> I’ve been given one such dump to look at to try and find the 32 bit
> stacks.
> Well I can do a !process 0 0 and see the 32 bit process there - fine. I
> can dump its thread stacks (lots of wow64cpu!blah at the bottom…
> okay…).
>
> 0: kd> .process fffffa80079e4b30
> Implicit process is now fffffa80079e4b30<br>&gt; 0: kd&gt; !wow64exts.sw<br>&gt; Switched to 32bit mode<br>&gt; 0: kd:x86&gt; k<br>&gt; ^ Current scope machine type mismatch error in 'k'<br>&gt; 0: kd:x86&gt; !wow64exts.k<br>&gt; The context is partially valid. Only x86 user-mode context is available.<br>&gt; Walking 64bit Stack...<br>&gt; *** Stack trace for last set context - .thread/.cxr resets it<br>&gt; Child-SP RetAddr Call Site<br>&gt; fffff88007740710 fffff80002ad5752 nt!KiSwapContext+0x7a<br>&gt; fffff88007740850 fffff80002ad78af nt!KiCommitThreadWait+0x1d2<br>&gt; fffff880077408e0 fffff80002aacfc8 nt!KeWaitForSingleObject+0x19f<br>&gt; fffff88007740980 fffff80002aab009 nt!KiSuspendThread+0x74<br>&gt; fffff880077409c0 fffff80002ad595d nt!KiDeliverApc+0x211<br>&gt; fffff88007740a40 fffff80002ad78af nt!KiCommitThreadWait+0x3dd<br>&gt; fffff88007740ad0 fffff80002dc9db2 nt!KeWaitForSingleObject+0x19f<br>&gt; fffff88007740b70 fffff80002acd853 nt!NtWaitForSingleObject+0xb2<br>&gt; fffff88007740be0 0000000074cb2dd9 nt!KiSystemServiceCopyEnd+0x13<br>&gt; 000000000008ebf8 0000000074cb2bcd 0x74cb2dd9<br>&gt; 000000000008ec00 0000000077220190 0x74cb2bcd<br>&gt; 000000000008ec08 0000000074d20023 0x77220190<br>&gt; 000000000008ec10 00000000002cc5a8 0x74d20023<br>&gt; 000000000008ec18 000000000018fff0 0x2cc5a8<br>&gt; 000000000008ec20 000000000008ec50 0x18fff0<br>&gt; 000000000008ec28 0000000000390008 0x8ec50<br>&gt; 000000000008ec30 00000000`00000000 0x390008
> Walking 32bit Stack…
> ^ Current scope machine type mismatch error in 'k '
> !wow64exts.k: Execute failed: 0x80040205
>
>
> ?? Current scope machine type mismatch error in 'k ’ ??
>
> I suspect there is some key fact I’m not aware of here.
>
> I did have a look in the windbg help but as I interpreted it, if you’re
> grabbing a full memory dump it doesn’t matter whether the machine
> running windbg - or the version of windbg being run - is 64 bit or not. So
> it must be something else - any help gratefully received.
>

Try .thread /w.

Thanks,
Pavel

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Scott Noone
Sent: Wednesday, August 25, 2010 11:34 AM
To: Kernel Debugging Interest List
Subject: Re:[windbg] wow64exts, or, where are the 32 bit stacks

OK, there’s certainly something wrong here. I’m not sure if this is a bug in the current implementation or a limitation. I’ll demonstrate what I’m seeing on a live dump as it ends better than the crash dump version :slight_smile:

I force a context switch to a 32bit process in WinDBG via .process /i, in this case I picked 32bit IE:

0: kd> !process -1 0
PROCESS fffffa8002455340
SessionId: 1 Cid: 0aa0 Peb: 7efdf000 ParentCid: 0444
DirBase: 1a2a3000 ObjectTable: fffff8a002e36e40 HandleCount: 327.
Image: iexplore.exe

I then switch to a thread in the process and attempt to do !wow64exts.k, which meets the same fate as yours:

0: kd> !wow64exts.k
Walking 64bit Stack…
*** Stack trace for last set context - .thread/.cxr resets it
Child-SP RetAddr Call Site
fffff8800473a660 fffff800026c2052 nt!KiSwapContext+0x7a
fffff8800473a7a0 fffff800026c41af nt!KiCommitThreadWait+0x1d2
fffff8800473a830 fffff800026d94ef nt!KeWaitForSingleObject+0x19f
fffff8800473a8d0 fffff800029be736 nt!AlpcpSignalAndWait+0x8f
fffff8800473a980 fffff800029bc6c0 nt!AlpcpReceiveSynchronousReply+0x46
fffff8800473a9e0 fffff800029b9c3b nt!AlpcpProcessSynchronousRequest+0x33d
fffff8800473ab00 fffff800026ba153 nt!NtAlpcSendWaitReceivePort+0x1ab
fffff8800473abb0 0000000077b6070a nt!KiSystemServiceCopyEnd+0x13
000000000371e758 000000007581e2f0 ntdll!ZwAlpcSendWaitReceivePort+0xa
000000000371e760 000000007580cf87 wow64!whNtAlpcSendWaitReceivePort+0x64
000000000371e7c0 000000007579276d wow64!Wow64SystemServiceEx+0xd7
000000000371f080 000000007580d07e
wow64cpu!TurboDispatchJumpAddressEnd+0x24
000000000371f140 000000007580c549 wow64!RunCpuSimulation+0xa
000000000371f190 0000000077b8d177 wow64!Wow64LdrpInitialize+0x429
000000000371f6e0 0000000077b4308e ntdll! ?? ::FNODOBFM::string'+0x2bfe4 000000000371f750 00000000`00000000 ntdll!LdrInitializeThunk+0xe Walking 32bit Stack…
^ Current scope machine type mismatch error in 'k ’
!wow64exts.k: Execute failed: 0x80040205

Thanks for the tip, that certainly seems to do the trick to get the user
mode stack for that thread. It’s all a bit hairy though, now !wow64exts.k
won’t show the x64 stack and getting back to the real stack isn’t overly
intuitive:

1: kd> .thread /w fffffa8001b00660
Implicit thread is now fffffa80`01b00660
x86 context set
1: kd:x86> !wow64exts.k
Walking 64bit Stack…
^ Current scope machine type mismatch error in 'k ’
!wow64exts.k: Execute failed: 0x80040205
1: kd:x86> kc
*** Stack trace for last set context - .thread/.cxr resets it

ntdll_77cf0000!NtRemoveIoCompletion
KERNELBASE!GetQueuedCompletionStatus
Dxtrans!TMThreadProc
kernel32!BaseThreadInitThunk
ntdll_77cf0000!__RtlUserThreadStart
ntdll_77cf0000!_RtlUserThreadStart
1: kd:x86> .thread fffffa8001b00660
Implicit thread is now fffffa80`01b00660
1: kd:x86> kc
*** Stack trace for last set context - .thread/.cxr resets it

WARNING: Frame IP not in any known module. Following frames may be wrong.
0x0
1: kd:x86> .effmach #
Effective machine: x64 (AMD64)
1: kd> kc
^ Current scope machine type mismatch error in ‘kc’
1: kd> .thread fffffa8001b00660
Implicit thread is now fffffa80`01b00660
1: kd> kc
*** Stack trace for last set context - .thread/.cxr resets it
Call Site
nt!KiSwapContext
nt!KiCommitThreadWait
nt!KeRemoveQueueEx
nt!IoRemoveIoCompletion
nt!NtRemoveIoCompletion
nt!KiSystemServiceCopyEnd
wow64cpu!CpupSyscallStub
wow64cpu!RemoveIoCompletionFault
wow64!RunCpuSimulation
wow64!Wow64LdrpInitialize
ntdll!_LdrpInitialize
ntdll!LdrInitializeThunk

So, the magic incantation to get back from .thread /w appears to be:

.effmach #
.thread

Is that correct?

Thanks!

-scott

“Pavel Lebedynskiy” wrote in message
news:xxxxx@windbg…
> Try .thread /w.
>
> Thanks,
> Pavel
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Scott Noone
> Sent: Wednesday, August 25, 2010 11:34 AM
> To: Kernel Debugging Interest List
> Subject: Re:[windbg] wow64exts, or, where are the 32 bit stacks
>
> OK, there’s certainly something wrong here. I’m not sure if this is a bug
> in the current implementation or a limitation. I’ll demonstrate what I’m
> seeing on a live dump as it ends better than the crash dump version :slight_smile:
>
> I force a context switch to a 32bit process in WinDBG via .process /i, in
> this case I picked 32bit IE:
>
> 0: kd> !process -1 0
> PROCESS fffffa8002455340
> SessionId: 1 Cid: 0aa0 Peb: 7efdf000 ParentCid: 0444
> DirBase: 1a2a3000 ObjectTable: fffff8a002e36e40 HandleCount: 327.
> Image: iexplore.exe
>
> I then switch to a thread in the process and attempt to do !wow64exts.k,
> which meets the same fate as yours:
>
> 0: kd> !wow64exts.k
> Walking 64bit Stack…
> *** Stack trace for last set context - .thread/.cxr resets it
> Child-SP RetAddr Call Site
> fffff8800473a660 fffff800026c2052 nt!KiSwapContext+0x7a
> fffff8800473a7a0 fffff800026c41af nt!KiCommitThreadWait+0x1d2
> fffff8800473a830 fffff800026d94ef nt!KeWaitForSingleObject+0x19f
> fffff8800473a8d0 fffff800029be736 nt!AlpcpSignalAndWait+0x8f
> fffff8800473a980 fffff800029bc6c0 nt!AlpcpReceiveSynchronousReply+0x46
> fffff8800473a9e0 fffff800029b9c3b
> nt!AlpcpProcessSynchronousRequest+0x33d
> fffff8800473ab00 fffff800026ba153 nt!NtAlpcSendWaitReceivePort+0x1ab
> fffff8800473abb0 0000000077b6070a nt!KiSystemServiceCopyEnd+0x13
> 000000000371e758 000000007581e2f0 ntdll!ZwAlpcSendWaitReceivePort+0xa
> 000000000371e760 000000007580cf87 wow64!whNtAlpcSendWaitReceivePort+0x64
> 000000000371e7c0 000000007579276d wow64!Wow64SystemServiceEx+0xd7
> 000000000371f080 000000007580d07e
> wow64cpu!TurboDispatchJumpAddressEnd+0x24
> 000000000371f140 000000007580c549 wow64!RunCpuSimulation+0xa
> 000000000371f190 0000000077b8d177 wow64!Wow64LdrpInitialize+0x429
> 000000000371f6e0 0000000077b4308e ntdll! ?? ::FNODOBFM::string'+0x2bfe4<br>&gt; 000000000371f750 00000000`00000000 ntdll!LdrInitializeThunk+0xe Walking
> 32bit Stack…
> ^ Current scope machine type mismatch error in 'k '
> !wow64exts.k: Execute failed: 0x80040205
>

I think that’s correct, but I’m not sure. It always takes me a few tries to get this to work.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Scott Noone
Sent: Friday, August 27, 2010 7:38 AM
To: Kernel Debugging Interest List
Subject: Re:[windbg] Re:wow64exts, or, where are the 32 bit stacks

So, the magic incantation to get back from .thread /w appears to be:

.effmach #
.thread

Is that correct?

OK, so long as it isn’t just me :slight_smile: This counts as my something new for the
day…

In the end, I guess it would be nice if !wow64exts.k was updated to hide all
of this from us.

-scott


Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

“Pavel Lebedynskiy” wrote in message
news:xxxxx@windbg…
> I think that’s correct, but I’m not sure. It always takes me a few tries
> to get this to work.
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Scott Noone
> Sent: Friday, August 27, 2010 7:38 AM
> To: Kernel Debugging Interest List
> Subject: Re:[windbg] Re:wow64exts, or, where are the 32 bit stacks
>
> So, the magic incantation to get back from .thread /w appears to be:
>
> .effmach #
> .thread
>
> Is that correct?
>

Aha!

kd> .thread blah
Implicit thread is now blah
kd> .thread /w blah
Implicit thread is now blah
x86 context set
… .and lo! a 32 bit call stack!

(Having previously
kd>.process /p /r blahblahprocaddress )

That’s brilliant - now I know how to coax the information out… oh happy day! Thanks Pavel.

… only seems to work on some threads though… hmmm…

I know this an old thread but this problem still happens all the time. When .thread /w doesn’t work, I do the following:
kd> .thread ETHREAD
kd> !teb
kd> dds StackLimit StackBase
The stack values are plucked from the TEB32 section.