BSOD when running Wireshark with Ndis IM driver installed

Hi, I have an Ndis Intermediate driver with a protocol edge and 2 Miniports above it that bind to upper layer protocols. The driver itself works fine and is able to send/receive packets. However, I notice an issue when I run the driver on XenServer and start Wireshark.

Everything seems fine and traffic seems to flow through my driver when Wireshark is not running. But the moment I attempt to start Wireshark, I observe a BSOD every single time.

Looking at the backtrace, I am unable to figure out what can be the exact cause of this issue. Also, I am able to capture packets using Wireshark in my local environment, but not when my Windows host is running on a particular XenServer.

The backtrace is as follows:

MODULE_NAME: xenvif

FAULTING_MODULE: fffff8035688b000 nt

DEBUG_FLR_IMAGE_TIMESTAMP: 577b672b

READ_ADDRESS: unable to get nt!MmSpecialPoolStart
unable to get nt!MmSpecialPoolEnd
unable to get nt!MmPagedPoolEnd
unable to get nt!MmNonPagedPoolStart
unable to get nt!MmSizeOfNonPagedPoolInBytes
0000000000000000

CURRENT_IRQL: 0

FAULTING_IP:
xenvif+f6e3
fffff801`afa4d6e3 498b1e mov rbx,qword ptr [r14]

DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT

BUGCHECK_STR: AV

ANALYSIS_VERSION: 6.3.9600.17237 (debuggers(dbg).140716-0327) amd64fre

LAST_CONTROL_TRANSFER: from fffff803569e4ee9 to fffff803569d93a0

STACK_TEXT:
ffffd0013e9e50d8 fffff803569e4ee9 : 000000000000000a 0000000000000000 0000000000000002 0000000000000000 : nt!KeBugCheckEx
ffffd0013e9e50e0 fffff803569e373a : 0000000000000000 0000000000000001 4138463534313500 754274656e202c30 : nt!setjmpex+0x37d9
ffffd0013e9e5220 fffff801afa4d6e3 : fffff801b20bab40 ffffe001000007c4 fffff801b20babb0 ffffe0015145f8a0 : nt!setjmpex+0x202a
ffffd0013e9e53b0 fffff801afa56ef0 : 0000000100000001 0000000000000001 ffffe0015191db48 ffffe0015191db30 : xenvif+0xf6e3
ffffd0013e9e5400 fffff801afb75e95 : fffffffffffffffe ffffe00150f60230 ffffd0013e9e5602 0000000000000000 : xenvif+0x18ef0
ffffd0013e9e5450 fffff801ae8630fd : ffffe00151843280 0000000000000000 0000000000000000 0000000000000003 : xennet+0x3e95
ffffd0013e9e5480 fffff801ae861f60 : ffffe00150783838 ffffd0013e9e5670 ffffe001519871a0 0000000000000000 : NDIS!NdisMIndicateReceiveNetBufferLists+0x117d
ffffd0013e9e54e0 fffff801ae862094 : ffffe001519871a0 ffffd0013e9e5501 00000000000075d9 59ccdb191d1b3762 : NDIS!NdisAcquireRWLockWrite+0xbb0
ffffd0013e9e5570 fffff801afb75dd2 : 0000000000000000 ffffd0013e9e5800 ffffe00150f60230 ffffd0013f535000 : NDIS!NdisMIndicateReceiveNetBufferLists+0x114
ffffd0013e9e5760 fffff801afb75345 : 0000000000000000 0000000000000000 ffffe00151843280 ffffe001519ab010 : xennet+0x3dd2
ffffd0013e9e57d0 fffff801afa4d34f : ffffe001518f56d0 ffffd00100000002 ffffd0013e9e5820 0000000000000000 : xennet+0x3345
ffffd0013e9e5800 fffff801afa488a0 : 0000000000000001 ffffe001518caab0 ffffe001518c9f50 ffffe001518caad0 : xenvif+0xf34f
ffffd0013e9e5850 fffff803568f5bf0 : ffffd0013e9bef00 ffffd0013e9e5990 ffffd0013e9e5b20 ffffd0013e9e5a20 : xenvif+0xa8a0
ffffd0013e9e5890 fffff803568f4f37 : ffffe0015175a000 ffffe0015175a080 0000000000000000 ffffd00100000001 : nt!ExAcquireRundownProtectionCacheAwareEx+0xfe0
ffffd0013e9e59e0 fffff803569dceea : ffffd0013e9bc180 ffffd0013e9bc180 ffffd0013e9c8240 ffffe00151655080 : nt!ExAcquireRundownProtectionCacheAwareEx+0x327
ffffd0013e9e5c60 0000000000000000 : ffffd0013e9e6000 ffffd0013e9e0000 0000000000000000 0000000000000000 : nt!KeSynchronizeExecution+0x2efa

I have also implemented all the required OIDs for my driver.

Thanks

Your symbols don’t look too together. Which makes me question if the stack dump is really valid. It seems possible the symbols for xenvif and xennet are available, but not on the Microsoft symbol servers.

Just offhand, if the symbols are valid, it looks like from your dump that you’re calling a bunch of functions while inside KeSynchronizeExecution, which typically raises the IRQL to a device interrupt level. Functions like NdisMIndicateReceiveNetBufferLists are specified to run <= DISPATCH_LEVEL

It seems odd that according to the stack dump, ExAcquireRundownProtectionCacheAwareEx and NdisMIndicateReceiveNetBufferLists are in the same call stack. I also could be its’ really executing on some code that has no symbols, and those are just the nearest symbols it can resolve.

I’d suggest you run things with DriverVerifier enabled, with a bunch of validation options enabled (not things like low resource simulation) and see if it’s detects some violation. I’d enable DriverVerifer for all drivers, or at least all the networking drivers that might be involved.

Jan

On 10/28/16, 12:22 PM, “xxxxx@lists.osr.com on behalf of xxxxx@gmail.com” wrote:

Hi, I have an Ndis Intermediate driver with a protocol edge and 2 Miniports above it that bind to upper layer protocols. The driver itself works fine and is able to send/receive packets. However, I notice an issue when I run the driver on XenServer and start Wireshark.

Everything seems fine and traffic seems to flow through my driver when Wireshark is not running. But the moment I attempt to start Wireshark, I observe a BSOD every single time.

Looking at the backtrace, I am unable to figure out what can be the exact cause of this issue. Also, I am able to capture packets using Wireshark in my local environment, but not when my Windows host is running on a particular XenServer.

The backtrace is as follows:

MODULE_NAME: xenvif

FAULTING_MODULE: fffff8035688b000 nt

DEBUG_FLR_IMAGE_TIMESTAMP: 577b672b

READ_ADDRESS: unable to get nt!MmSpecialPoolStart
unable to get nt!MmSpecialPoolEnd
unable to get nt!MmPagedPoolEnd
unable to get nt!MmNonPagedPoolStart
unable to get nt!MmSizeOfNonPagedPoolInBytes
0000000000000000

CURRENT_IRQL: 0

FAULTING_IP:
xenvif+f6e3
fffff801afa4d6e3 498b1e mov rbx,qword ptr [r14]<br><br>DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT<br><br>BUGCHECK_STR: AV<br><br>ANALYSIS_VERSION: 6.3.9600.17237 (debuggers(dbg).140716-0327) amd64fre<br><br>LAST_CONTROL_TRANSFER: from fffff803569e4ee9 to fffff803569d93a0<br><br>STACK_TEXT: <br> ffffd0013e9e50d8 fffff803569e4ee9 : 000000000000000a 0000000000000000 0000000000000002 0000000000000000 : nt!KeBugCheckEx<br> ffffd0013e9e50e0 fffff803569e373a : 0000000000000000 0000000000000001 4138463534313500 754274656e202c30 : nt!setjmpex+0x37d9<br> ffffd0013e9e5220 fffff801afa4d6e3 : fffff801b20bab40 ffffe001000007c4 fffff801b20babb0 ffffe0015145f8a0 : nt!setjmpex+0x202a<br> ffffd0013e9e53b0 fffff801afa56ef0 : 0000000100000001 0000000000000001 ffffe0015191db48 ffffe0015191db30 : xenvif+0xf6e3<br> ffffd0013e9e5400 fffff801afb75e95 : fffffffffffffffe ffffe00150f60230 ffffd0013e9e5602 0000000000000000 : xenvif+0x18ef0<br> ffffd0013e9e5450 fffff801ae8630fd : ffffe00151843280 0000000000000000 0000000000000000 0000000000000003 : xennet+0x3e95<br> ffffd0013e9e5480 fffff801ae861f60 : ffffe00150783838 ffffd0013e9e5670 ffffe001519871a0 0000000000000000 : NDIS!NdisMIndicateReceiveNetBufferLists+0x117d<br> ffffd0013e9e54e0 fffff801ae862094 : ffffe001519871a0 ffffd0013e9e5501 00000000000075d9 59ccdb191d1b3762 : NDIS!NdisAcquireRWLockWrite+0xbb0<br> ffffd0013e9e5570 fffff801afb75dd2 : 0000000000000000 ffffd0013e9e5800 ffffe00150f60230 ffffd0013f535000 : NDIS!NdisMIndicateReceiveNetBufferLists+0x114<br> ffffd0013e9e5760 fffff801afb75345 : 0000000000000000 0000000000000000 ffffe00151843280 ffffe001519ab010 : xennet+0x3dd2<br> ffffd0013e9e57d0 fffff801afa4d34f : ffffe001518f56d0 ffffd00100000002 ffffd0013e9e5820 0000000000000000 : xennet+0x3345<br> ffffd0013e9e5800 fffff801afa488a0 : 0000000000000001 ffffe001518caab0 ffffe001518c9f50 ffffe001518caad0 : xenvif+0xf34f<br> ffffd0013e9e5850 fffff803568f5bf0 : ffffd0013e9bef00 ffffd0013e9e5990 ffffd0013e9e5b20 ffffd0013e9e5a20 : xenvif+0xa8a0<br> ffffd0013e9e5890 fffff803568f4f37 : ffffe0015175a000 ffffe0015175a080 0000000000000000 ffffd00100000001 : nt!ExAcquireRundownProtectionCacheAwareEx+0xfe0<br> ffffd0013e9e59e0 fffff803569dceea : ffffd0013e9bc180 ffffd0013e9bc180 ffffd0013e9c8240 ffffe00151655080 : nt!ExAcquireRundownProtectionCacheAwareEx+0x327<br> ffffd0013e9e5c60 0000000000000000 : ffffd0013e9e6000 ffffd0013e9e0000 0000000000000000 00000000`00000000 : nt!KeSynchronizeExecution+0x2efa

I have also implemented all the required OIDs for my driver.

Thanks


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:>

Thanks Jan. I am however not able to figure out why starting Wireshark would result in a crash. That seems to be my primary concern. Without Wireshark, my Tx and Rx paths seem to work just fine.

My protocol edge is set to be in promiscuous mode already (before starting Wireshark).
There is clearly something happening on starting Wireshark that causes the crash, and this seems to be only happening when my driver is installed.

It’s not uncommon to have bad interactions between components. For example, the OS may cope with NBL chains that are not quite correct, and wireshark might choke on those packets. To give a concrete example, an NB has an offset field, which the OS is happy to use when receiving a packet. If you run the Microsoft certification tests for a NIC miniport, the certifications tests do not correctly handle packets with a non-zero offset for one of the tests and what seems like an unrelated test fails. I believe it was the test that verifies Tx checksum generation.

What you are seeing might be a bug in the wireshark capture driver, but then again it might be a bug in your driver. Since the number of operational hours on wireshark is large, and the number of operational hours on your driver is likely small, chances seem much higher it’s a bug in your driver. The bottom line is a failure does not happen when using wireshark without your driver, and with your driver installed the system crashes. Even if the root cause is a bug in the wireshark capture driver, your driver will likely get blamed, as it’s the delta between a system that works and a system that doesn’t. Actually, have you verified that wireshark without your driver works fine under Xen?

As an experiment, you might see if a crash also occurs when using the Microsoft network monitor. If so, that adds evidence your driver has the bug. I’d also suggest disabling some of the other protocol bindings, and see of that makes a difference. I’d also suggest changing the underlying NIC, and see if that makes a difference. I assume Xen has some hardware emulation NIC, and has synthetic NICs, and also believe you can pass though physical NICs to a VM.

There also is the issue of what the documentation says and what systems actually do. For example, the NDIS docs all say when transmitting an LSO packet there will only be a single NB attached to the NBL. If you run the driver certifications tests, during one test, NBLs with LSO enabled and multiple NB are transmitted. Even if you driver perfectly follows the docs, that’s no guarantee it will work correctly in the real world. Has you driver been running in the world for a year, and all of sudden, you are getting reports of crashes. Or has you driver never seen production use, so still is getting the initial bugs worked out.

What I’m getting at here is YOU have control over YOUR driver, and you don’t have a lot of control over other components, from Microsoft or other ISVs, so if a crash happens due to an interaction, you can file a bug on the other component, but you really need to fix the thing you have control over, which is your driver. I recently did a NIC minport driver, and there are I believe at least three workarounds to bugs in the Microsoft certifications tests. Had I sat around and said I could not pass the tests until Microsoft fixed their bugs, my guess is I’d be waiting a long time. In the end, it was just a lot faster for me to adjust my driver so it didn’t stimulate the bugs in other components.

My suggestions is you figure out why your driver chokes with wireshark when running under Xen, and don’t do whatever it doesn’t like. I’m guessing the Xen and wireshark captures driver are all open source, which makes your job of finding the interaction issue easier, as you can debug with full symbols and source. I’d suggest you make it a priority to track down the symbols and source for those other components, as the crash dump you supplied does not really have especially useful data. Unless you’re really skilled at psychic debugging, having real data about a crash helps a lot.

Did you take my suggestion and enable DriverVerifier? Has your driver passed the Microsoft certifications tests? Those tests aren’t perfect, but they do cover a wide range of areas, and you can run many of the NDIS tests manually from a test GUI (under Xen and wireshark). I assume you could manually run most of the NDIS NIC tests on your IM virtual NICs.

Jan

On 10/28/16, 7:45 PM, “xxxxx@lists.osr.com on behalf of xxxxx@gmail.com” wrote:

Thanks Jan. I am however not able to figure out why starting Wireshark would result in a crash. That seems to be my primary concern. Without Wireshark, my Tx and Rx paths seem to work just fine.

My protocol edge is set to be in promiscuous mode already (before starting Wireshark).
There is clearly something happening on starting Wireshark that causes the crash, and this seems to be only happening when my driver is installed.


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:>

Thanks Jan for your input. I enabled driver verifier and didn’t see any bugcheck. I think this has got to do something related to Wireshark issuing an OID request followed by looping back packets in software. I am not sure if this is what is happening.

When I start Wireshark, I do see NDIS_STATUS_PACKET_FILTER status indications on my protocol edge.

I also validated the packet’s offset field and it was 0. I know for a fact that Wireshark tries to put the interfaces into promiscuous mode. Maybe that is causing the issue? My protocol edge is already in promiscous mode.

If I just call NdisReturnNetBufferLists from PtReceiveNBL as if the packetFilter on the protocol edge were set to 0, I don’t see this crash. Clearly something on the receive path seems to be the cause. What I fail to understand is why does only starting Wireshark manifest this crash.

I got hold of the xenvif symbols and see the following stack trace:

xenvif!ReceiverReturnPacket+37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321]
fffff801`afa4d6e3 498b1e mov rbx,qword ptr [r14]

FAULTING_SOURCE_LINE: c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c

FAULTING_SOURCE_FILE: c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c

FAULTING_SOURCE_LINE_NUMBER: 3321

SYMBOL_STACK_INDEX: 3

SYMBOL_NAME: xenvif!ReceiverReturnPacket+37

FOLLOWUP_NAME: MachineOwner

IMAGE_NAME: xenvif.sys

BUCKET_ID: WRONG_SYMBOLS

FAILURE_BUCKET_ID: WRONG_SYMBOLS

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:wrong_symbols

FAILURE_ID_HASH: {70b057e8-2462-896f-28e7-ac72d4d365f8}

Followup: MachineOwner

1: kd> u fffff801afa4d6e3498b1e
01afa4d6`e3498b1e ?? ???
^ Memory access error in ‘u fffff801afa4d6e3498b1e’
1: kd> kn

Child-SP RetAddr Call Site

00 ffffd0013e9e50d8 fffff803569e4ee9 nt!KeBugCheckEx
01 ffffd0013e9e50e0 fffff803569e373a nt!setjmpex+0x37d9
02 ffffd0013e9e5220 fffff801afa4d6e3 nt!setjmpex+0x202a
03 (Inline Function) ---------------- xenvif!__RingReturnPacket+0x13 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 1386] 04 ffffd0013e9e53b0 fffff801afa56ef0 xenvif!ReceiverReturnPacket+0x37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321] 05 ffffd0013e9e5400 fffff801afb75e95 xenvif!VifReturnPacket+0x108 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\vif.c @ 385] 06 ffffd0013e9e5450 fffff801ae8630fd xennet+0x3e95 07 ffffd0013e9e5480 fffff801ae861f60 NDIS!NdisMIndicateReceiveNetBufferLists+0x117d 08 ffffd0013e9e54e0 fffff801ae862094 NDIS!NdisAcquireRWLockWrite+0xbb0 09 ffffd0013e9e5570 fffff801afb75dd2 NDIS!NdisMIndicateReceiveNetBufferLists+0x114 0a ffffd0013e9e5760 fffff801afb75345 xennet+0x3dd2 0b ffffd0013e9e57d0 fffff801afa4d34f xennet+0x3345 0c (Inline Function) ---------------- xenvif!__RingReleaseLock+0x6a [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 1332]
0d (Inline Function) ---------------- xenvif!__RingNotify+0x7c [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 2669] 0e ffffd0013e9e5800 fffff801afa488a0 xenvif!ReceiverNotify+0xa7 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3371] 0f ffffd0013e9e5850 fffff803568f5bf0 xenvif!NotifierDpc+0x74 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\notifier.c @ 159] 10 ffffd0013e9e5890 fffff803568f4f37 nt!ExAcquireRundownProtectionCacheAwareEx+0xfe0 11 ffffd0013e9e59e0 fffff803569dceea nt!ExAcquireRundownProtectionCacheAwareEx+0x327 12 ffffd0013e9e5c60 0000000000000000 nt!KeSynchronizeExecution+0x2efa 1: kd\> .frame 02 02 ffffd0013e9e5220 fffff801afa4d6e3 nt!setjmpex+0x202a 1: kd\> .frame 03 03 (Inline Function) ---------------- xenvif!__RingReturnPacket+0x13 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 1386]

Wireshark typically enables promiscuous mode which results in sends being looped back synchinously as receives and that can gobble up quite a bit of stack if filters are involved.

When I would see a symptom reported with Wireshark I generally suspect this sort of thing.

Wireshark has an advanced capture option to disable promiscuous mode but the issue is loopback. It will always enable the all local mode to see tx frames.

I would look at you stack consumption and handling of looped back tx frames.

Good luck.
Dave Cattley

Sent from my Windows 10 phone

xxxxx@gmail.com wrote:

I got hold of the xenvif symbols and see the following stack trace:

xenvif!ReceiverReturnPacket+37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321]
fffff801afa4d6e3 498b1e mov rbx,qword ptr [r14] ... 1: kd\> u fffff801afa4d6e3498b1e 01afa4d6e3498b1e ?? ???
^ Memory access error in ‘u fffff801afa4d6e3498b1e’

In case you didn’t see what you did there, the address is
fffff801afa4d6e3. The “498b1e” is the instruction at that address.

1: kd> kn

Child-SP RetAddr Call Site

00 ffffd0013e9e50d8 fffff803569e4ee9 nt!KeBugCheckEx
01 ffffd0013e9e50e0 fffff803569e373a nt!setjmpex+0x37d9
02 ffffd0013e9e5220 fffff801afa4d6e3 nt!setjmpex+0x202a
03 (Inline Function) ---------------- xenvif!__RingReturnPacket+0x13 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 1386] 04 ffffd0013e9e53b0 fffff801`afa56ef0 xenvif!ReceiverReturnPacket+0x37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321]

What happens at line 1386? The instruction says you are dereferencing a
null pointer.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Thanks David and Tim.

I do not have private symbols for xenvif, and am just looking through the source code to see what could be going wrong, although the public symbols I have might not match line to line with the source.

Here is a more detailed debugging info.

MODULE_NAME: xenvif

FAULTING_MODULE: fffff8035688b000 nt

DEBUG_FLR_IMAGE_TIMESTAMP: 577b672b

READ_ADDRESS: 0000000000000000

CURRENT_IRQL: 0

FAULTING_IP:
xenvif!ReceiverReturnPacket+37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321]
fffff801`afa4d6e3 498b1e mov rbx,qword ptr [r14]

DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT

BUGCHECK_STR: AV

ANALYSIS_VERSION: 6.3.9600.17237 (debuggers(dbg).140716-0327) amd64fre

LAST_CONTROL_TRANSFER: from fffff803569e4ee9 to fffff803569d93a0

STACK_TEXT:
ffffd0013e9e50d8 fffff803569e4ee9 : 000000000000000a 0000000000000000 0000000000000002 0000000000000000 : nt!KeBugCheckEx
ffffd0013e9e50e0 fffff803569e373a : 0000000000000000 0000000000000001 4138463534313500 754274656e202c30 : nt!setjmpex+0x37d9
ffffd0013e9e5220 fffff801afa4d6e3 : fffff801b20bab40 ffffe001000007c4 fffff801b20babb0 ffffe0015145f8a0 : nt!setjmpex+0x202a
ffffd0013e9e53b0 fffff801afa56ef0 : 0000000100000001 0000000000000001 ffffe0015191db48 ffffe0015191db30 : xenvif!ReceiverReturnPacket+0x37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321]
ffffd0013e9e5400 fffff801afb75e95 : fffffffffffffffe ffffe00150f60230 ffffd0013e9e5602 0000000000000000 : xenvif!VifReturnPacket+0x108 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\vif.c @ 385]
ffffd0013e9e5450 fffff801ae8630fd : ffffe00151843280 0000000000000000 0000000000000000 0000000000000003 : xennet+0x3e95
ffffd0013e9e5480 fffff801ae861f60 : ffffe00150783838 ffffd0013e9e5670 ffffe001519871a0 0000000000000000 : NDIS!NdisMIndicateReceiveNetBufferLists+0x117d
ffffd0013e9e54e0 fffff801ae862094 : ffffe001519871a0 ffffd0013e9e5501 00000000000075d9 59ccdb191d1b3762 : NDIS!NdisAcquireRWLockWrite+0xbb0
ffffd0013e9e5570 fffff801afb75dd2 : 0000000000000000 ffffd0013e9e5800 ffffe00150f60230 ffffd0013f535000 : NDIS!NdisMIndicateReceiveNetBufferLists+0x114
ffffd0013e9e5760 fffff801afb75345 : 0000000000000000 0000000000000000 ffffe00151843280 ffffe001519ab010 : xennet+0x3dd2
ffffd0013e9e57d0 fffff801afa4d34f : ffffe001518f56d0 ffffd00100000002 ffffd0013e9e5820 0000000000000000 : xennet+0x3345
ffffd0013e9e5800 fffff801afa488a0 : 0000000000000001 ffffe001518caab0 ffffe001518c9f50 ffffe001518caad0 : xenvif!ReceiverNotify+0xa7 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3371]
ffffd0013e9e5850 fffff803568f5bf0 : ffffd0013e9bef00 ffffd0013e9e5990 ffffd0013e9e5b20 ffffd0013e9e5a20 : xenvif!NotifierDpc+0x74 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\notifier.c @ 159]
ffffd0013e9e5890 fffff803568f4f37 : ffffe0015175a000 ffffe0015175a080 0000000000000000 ffffd00100000001 : nt!ExAcquireRundownProtectionCacheAwareEx+0xfe0
ffffd0013e9e59e0 fffff803569dceea : ffffd0013e9bc180 ffffd0013e9bc180 ffffd0013e9c8240 ffffe00151655080 : nt!ExAcquireRundownProtectionCacheAwareEx+0x327
ffffd0013e9e5c60 0000000000000000 : ffffd0013e9e6000 ffffd0013e9e0000 0000000000000000 0000000000000000 : nt!KeSynchronizeExecution+0x2efa

STACK_COMMAND: kb

FOLLOWUP_IP:
xenvif!ReceiverReturnPacket+37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321]
fffff801`afa4d6e3 498b1e mov rbx,qword ptr [r14]

FAULTING_SOURCE_LINE: c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c

FAULTING_SOURCE_FILE: c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c

FAULTING_SOURCE_LINE_NUMBER: 3321

SYMBOL_STACK_INDEX: 3

SYMBOL_NAME: xenvif!ReceiverReturnPacket+37

FOLLOWUP_NAME: MachineOwner

IMAGE_NAME: xenvif.sys

BUCKET_ID: WRONG_SYMBOLS

FAILURE_BUCKET_ID: WRONG_SYMBOLS

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:wrong_symbols

FAILURE_ID_HASH: {70b057e8-2462-896f-28e7-ac72d4d365f8}

Followup: MachineOwner

1: kd> kn

Child-SP RetAddr Call Site

00 ffffd0013e9e50d8 fffff803569e4ee9 nt!KeBugCheckEx
01 ffffd0013e9e50e0 fffff803569e373a nt!setjmpex+0x37d9
02 ffffd0013e9e5220 fffff801afa4d6e3 nt!setjmpex+0x202a
03 (Inline Function) ---------------- xenvif!__RingReturnPacket+0x13 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 1386] 04 ffffd0013e9e53b0 fffff801afa56ef0 xenvif!ReceiverReturnPacket+0x37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321] 05 ffffd0013e9e5400 fffff801afb75e95 xenvif!VifReturnPacket+0x108 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\vif.c @ 385] 06 ffffd0013e9e5450 fffff801ae8630fd xennet+0x3e95 07 ffffd0013e9e5480 fffff801ae861f60 NDIS!NdisMIndicateReceiveNetBufferLists+0x117d 08 ffffd0013e9e54e0 fffff801ae862094 NDIS!NdisAcquireRWLockWrite+0xbb0 09 ffffd0013e9e5570 fffff801afb75dd2 NDIS!NdisMIndicateReceiveNetBufferLists+0x114 0a ffffd0013e9e5760 fffff801afb75345 xennet+0x3dd2 0b ffffd0013e9e57d0 fffff801afa4d34f xennet+0x3345 0c (Inline Function) ---------------- xenvif!__RingReleaseLock+0x6a [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 1332]
0d (Inline Function) ---------------- xenvif!__RingNotify+0x7c [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 2669] 0e ffffd0013e9e5800 fffff801afa488a0 xenvif!ReceiverNotify+0xa7 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3371] 0f ffffd0013e9e5850 fffff803568f5bf0 xenvif!NotifierDpc+0x74 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\notifier.c @ 159] 10 ffffd0013e9e5890 fffff803568f4f37 nt!ExAcquireRundownProtectionCacheAwareEx+0xfe0 11 ffffd0013e9e59e0 fffff803569dceea nt!ExAcquireRundownProtectionCacheAwareEx+0x327 12 ffffd0013e9e5c60 00000000`00000000 nt!KeSynchronizeExecution+0x2efa

1: kd> ub fffff801afa4d6e3
xenvif!ReceiverReturnPacket+0x1b [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3318]:
fffff801afa4d6c7 4c8b7260 mov r14,qword ptr [rdx+60h] fffff801afa4d6cb 4883626000 and qword ptr [rdx+60h],0
fffff801afa4d6d0 488d7a68 lea rdi,[rdx+68h] fffff801afa4d6d4 4c8bf9 mov r15,rcx
fffff801afa4d6d7 4885ff test rdi,rdi fffff801afa4d6da 744a je xenvif!ReceiverReturnPacket+0x7a (fffff801afa4d726) fffff801afa4d6dc 488b37 mov rsi,qword ptr [rdi]
fffff801`afa4d6df 48832700 and qword ptr [rdi],0

1: kd> dv /v
Unable to enumerate locals, Win32 error 0n318
Private symbols (symbols.pri) are required for locals.
Type “.hh dbgerr005” for details.

1: kd> .frame /r 03
03 (Inline Function) ---------------- xenvif!__RingReturnPacket+0x13 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 1386] rax=ffffe001518f56d0 rbx=0000000000000001 rcx=ffffe001518f56d0 rdx=ffffe0015178f618 rsi=0000000000000000 rdi=ffffe0015178f680 rip=fffff801afa4d6e3 rsp=ffffd0013e9e53b0 rbp=ffffd0013e9e53e0 r8=0000000000000000 r9=0000000000000000 r10=ffffd0013e9c8240 r11=ffffe001518c9f50 r12=0000000000000002 r13=0000000000000003 r14=0000000000000000 r15=ffffe001518f56d0 iopl=0 nv up ei ng nz na pe nc cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00000282 xenvif!__RingReturnPacket+0x13 [inlined in xenvif!ReceiverReturnPacket+0x37]: fffff801afa4d6e3 498b1e mov rbx,qword ptr [r14] ds:002b:00000000`00000000=???

1: kd> ub fffff801afa4d6e3
xenvif!ReceiverReturnPacket+0x1b [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3318]:
fffff801afa4d6c7 4c8b7260 mov r14,qword ptr [rdx+60h] fffff801afa4d6cb 4883626000 and qword ptr [rdx+60h],0
fffff801afa4d6d0 488d7a68 lea rdi,[rdx+68h] fffff801afa4d6d4 4c8bf9 mov r15,rcx
fffff801afa4d6d7 4885ff test rdi,rdi fffff801afa4d6da 744a je xenvif!ReceiverReturnPacket+0x7a (fffff801afa4d726) fffff801afa4d6dc 488b37 mov rsi,qword ptr [rdi]
fffff801`afa4d6df 48832700 and qword ptr [rdi],0

So it seems like somewhere a NULL pointer is being dereferenced.

Show a bit of different info now:

FOLLOWUP_IP:
xenvif!ReceiverReturnPacket+37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321]
fffff801`afa4d6e3 498b1e mov rbx,qword ptr [r14]

FAULTING_SOURCE_LINE: c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c

FAULTING_SOURCE_FILE: c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c

FAULTING_SOURCE_LINE_NUMBER: 3321

SYMBOL_STACK_INDEX: 3

SYMBOL_NAME: xenvif!ReceiverReturnPacket+37

FOLLOWUP_NAME: MachineOwner

IMAGE_NAME: xenvif.sys

BUCKET_ID: WRONG_SYMBOLS

FAILURE_BUCKET_ID: WRONG_SYMBOLS

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:wrong_symbols

FAILURE_ID_HASH: {70b057e8-2462-896f-28e7-ac72d4d365f8}

Followup: MachineOwner

1: kd> u fffff801afa4d6e3
xenvif!__RingPutPacket [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321] [inlined in xenvif!ReceiverReturnPacket+0x37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321]]:
fffff801afa4d6e3 498b1e mov rbx,qword ptr [r14] fffff801afa4d6e6 33d2 xor edx,edx
fffff801afa4d6e8 488d4f98 lea rcx,[rdi-68h] fffff801afa4d6ec 448d4268 lea r8d,[rdx+68h]
fffff801afa4d6f0 e8abbc0000 call xenvif!memset (fffff801afa593a0)
fffff801afa4d6f5 83672c00 and dword ptr [rdi+2Ch],0 fffff801afa4d6f9 83672800 and dword ptr [rdi+28h],0
fffff801afa4d6fd 488b4720 mov rax,qword ptr [rdi+20h] 1: kd\> .frame /r 03 03 (Inline Function) ---------------- xenvif!__RingReturnPacket+0x13 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 1386]
rax=ffffe001518f56d0 rbx=0000000000000001 rcx=ffffe001518f56d0
rdx=ffffe0015178f618 rsi=0000000000000000 rdi=ffffe0015178f680
rip=fffff801afa4d6e3 rsp=ffffd0013e9e53b0 rbp=ffffd0013e9e53e0
r8=0000000000000000 r9=0000000000000000 r10=ffffd0013e9c8240
r11=ffffe001518c9f50 r12=0000000000000002 r13=0000000000000003
r14=0000000000000000 r15=ffffe001518f56d0
iopl=0 nv up ei ng nz na pe nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00000282
xenvif!__RingReturnPacket+0x13 [inlined in xenvif!ReceiverReturnPacket+0x37]:
fffff801afa4d6e3 498b1e mov rbx,qword ptr [r14] ds:002b:0000000000000000=???
1: kd> dv
Mdl = 0xffffe0015178f680 Next = 0x0000000000000000
Frontend =
1: kd> dt Mdl
Local var @ rdi Type _MDL*
+0x000 Next : (null)
+0x008 Size : 0n56
+0x00a MdlFlags : 0n3
+0x010 Process : (null)
+0x018 MappedSystemVa : 0xffffd00140134000 Void<br> +0x020 StartVa : 0xffffd00140134000 Void
+0x028 ByteCount : 0
+0x02c ByteOffset : 0

The symbols seemed to be messed up earlier. Here is a more accurate trace:

SYMBOL_STACK_INDEX: 3

SYMBOL_NAME: xenvif!ReceiverReturnPacket+37

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: xenvif

IMAGE_NAME: xenvif.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 577b672b

BUCKET_ID_FUNC_OFFSET: 37

FAILURE_BUCKET_ID: AV_xenvif!ReceiverReturnPacket

BUCKET_ID: AV_xenvif!ReceiverReturnPacket

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:av_xenvif!receiverreturnpacket

FAILURE_ID_HASH: {0c093ea2-c479-d425-20c9-41e400fd1dca}

Followup: MachineOwner

1: kd> kn

Child-SP RetAddr Call Site

00 ffffd0013e9e50d8 fffff803569e4ee9 nt!KeBugCheckEx
01 ffffd0013e9e50e0 fffff803569e373a nt!KiBugCheckDispatch+0x69
02 ffffd0013e9e5220 fffff801afa4d6e3 nt!KiPageFault+0x23a
03 (Inline Function) ---------------- xenvif!__RingReturnPacket+0x13 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 1386] 04 ffffd0013e9e53b0 fffff801afa56ef0 xenvif!ReceiverReturnPacket+0x37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321] 05 ffffd0013e9e5400 fffff801afb75e95 xenvif!VifReturnPacket+0x108 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\vif.c @ 385] 06 (Inline Function) ---------------- xennet!__ReceiverReturnNetBufferLists+0x3e [c:\jenkins\jobs\beta-build-xennet\workspace\src\xennet\receiver.c @ 248]
07 ffffd0013e9e5450 fffff801ae8630fd xennet!ReceiverReturnNetBufferLists+0x4d [c:\jenkins\jobs\beta-build-xennet\workspace\src\xennet\receiver.c @ 268]
08 ffffd0013e9e5480 fffff801ae861f60 NDIS!ndisReturnNetBufferListsInternal+0x12d
09 ffffd0013e9e54e0 fffff801ae862094 NDIS!ndisMTopReceiveNetBufferLists+0x30e
0a ffffd0013e9e5570 fffff801afb75dd2 NDIS!NdisMIndicateReceiveNetBufferLists+0x114
0b (Inline Function) ---------------- xennet!ReceiverPushPackets+0x72 [c:\jenkins\jobs\beta-build-xennet\workspace\src\xennet\receiver.c @ 373] 0c ffffd0013e9e5760 fffff801afb75345 xennet!ReceiverReceivePackets+0x1e6 [c:\jenkins\jobs\beta-build-xennet\workspace\src\xennet\receiver.c @ 443] 0d ffffd0013e9e57d0 fffff801afa4d34f xennet!AdapterVifCallback+0x35 [c:\jenkins\jobs\beta-build-xennet\workspace\src\xennet\adapter.c @ 372] 0e (Inline Function) ---------------- xenvif!__RingReleaseLock+0x6a [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 1332]
0f (Inline Function) ---------------- xenvif!__RingNotify+0x7c [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 2669] 10 ffffd0013e9e5800 fffff801afa488a0 xenvif!ReceiverNotify+0xa7 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3371] 11 ffffd0013e9e5850 fffff803568f5bf0 xenvif!NotifierDpc+0x74 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\notifier.c @ 159] 12 ffffd0013e9e5890 fffff803568f4f37 nt!KiExecuteAllDpcs+0x1b0 13 ffffd0013e9e59e0 fffff803569dceea nt!KiRetireDpcList+0xd7 14 ffffd0013e9e5c60 00000000`00000000 nt!KiIdleLoop+0x5a

1: kd> .frame 3
03 (Inline Function) ---------------- xenvif!__RingReturnPacket+0x13 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 1386] 1: kd\> r rax=ffffd0013e9e51e0 rbx=0000000000000001 rcx=000000000000000a rdx=0000000000000000 rsi=0000000000000000 rdi=ffffe0015178f680 rip=fffff803569d93a0 rsp=ffffd0013e9e50d8 rbp=ffffd0013e9e52a0 r8=0000000000000002 r9=0000000000000000 r10=fffff801afa4d6e3 r11=ffffd0013e9e5198 r12=0000000000000002 r13=0000000000000003 r14=0000000000000000 r15=ffffe001518f56d0 iopl=0 nv up ei ng nz na pe nc cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00000282 nt!KeBugCheckEx: fffff803569d93a0 48894c2408 mov qword ptr [rsp+8],rcx ss:0018:ffffd0013e9e50e0=000000000000000a 1: kd\> .frame /r 3 03 (Inline Function) ---------------- xenvif!__RingReturnPacket+0x13 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 1386]
rax=ffffe001518f56d0 rbx=0000000000000001 rcx=ffffe001518f56d0
rdx=ffffe0015178f618 rsi=0000000000000000 rdi=ffffe0015178f680
rip=fffff801afa4d6e3 rsp=ffffd0013e9e53b0 rbp=ffffd0013e9e53e0
r8=0000000000000000 r9=0000000000000000 r10=ffffd0013e9c8240
r11=ffffe001518c9f50 r12=0000000000000002 r13=0000000000000003
r14=0000000000000000 r15=ffffe001518f56d0
iopl=0 nv up ei ng nz na pe nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00000282
xenvif!__RingReturnPacket+0x13 [inlined in xenvif!ReceiverReturnPacket+0x37]:
fffff801afa4d6e3 498b1e mov rbx,qword ptr [r14] ds:002b:0000000000000000=???

1: kd> .frame 4
04 ffffd0013e9e53b0 fffff801afa56ef0 xenvif!ReceiverReturnPacket+0x37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321]
1: kd> dv /V
@r15 @r15 Receiver = 0xffffe001518f56d0 @rdx @rdx Packet = 0xffffe0015178f618
@r14 @r14 Ring = 0x0000000000000000 1: kd\> dt Packet Local var @ rdx Type _XENVIF_RECEIVER_PACKET\* +0x000 ListEntry : _LIST_ENTRY [0x0000000000000000 - 0x00000000`00000000]
+0x010 Offset : 0
+0x014 Length : 0
+0x018 Info : _XENVIF_PACKET_INFO
+0x058 Flags : _XENVIF_CHECKSUM_FLAGS
+0x05c TagControlInformation : 0
+0x05e MaximumSegmentSize : 0
+0x060 Cookie : (null)
+0x068 Mdl : _MDL
+0x098 __Pfn : 0x22f52

1: kd> dt -r Packet
Local var @ rdx Type _XENVIF_RECEIVER_PACKET*
+0x000 ListEntry : _LIST_ENTRY [0x0000000000000000 - 0x0000000000000000]
+0x000 Flink : (null)
+0x008 Blink : (null)
+0x010 Offset : 0
+0x014 Length : 0
+0x018 Info : _XENVIF_PACKET_INFO
+0x000 EthernetHeader : _XENVIF_PACKET_HEADER
+0x000 Offset : 0
+0x004 Length : 0
+0x008 LLCSnapHeader : _XENVIF_PACKET_HEADER
+0x000 Offset : 0
+0x004 Length : 0
+0x010 IpHeader : _XENVIF_PACKET_HEADER
+0x000 Offset : 0
+0x004 Length : 0
+0x018 IpOptions : _XENVIF_PACKET_HEADER
+0x000 Offset : 0
+0x004 Length : 0
+0x020 TcpHeader : _XENVIF_PACKET_HEADER
+0x000 Offset : 0
+0x004 Length : 0
+0x028 TcpOptions : _XENVIF_PACKET_HEADER
+0x000 Offset : 0
+0x004 Length : 0
+0x030 UdpHeader : _XENVIF_PACKET_HEADER
+0x000 Offset : 0
+0x004 Length : 0
+0x038 Flags : _XENVIF_PACKET_FLAGS
+0x000 IsAFragment : 0y0
+0x000 Reserved : 0y0000000000000000000000000000000 (0)
+0x03c Length : 0
+0x058 Flags : _XENVIF_CHECKSUM_FLAGS
+0x000 IpChecksumSucceeded : 0y0
+0x000 IpChecksumFailed : 0y0
+0x000 IpChecksumPresent : 0y0
+0x000 TcpChecksumSucceeded : 0y0
+0x000 TcpChecksumFailed : 0y0
+0x000 TcpChecksumPresent : 0y0
+0x000 UdpChecksumSucceeded : 0y0
+0x000 UdpChecksumFailed : 0y0
+0x000 UdpChecksumPresent : 0y0
+0x000 Reserved : 0y00000000000000000000000 (0)
+0x05c TagControlInformation : 0
+0x05e MaximumSegmentSize : 0
+0x060 Cookie : (null)
+0x068 Mdl : _MDL
+0x000 Next : (null)
+0x008 Size : 0n56
+0x00a MdlFlags : 0n3
+0x010 Process : (null)
+0x018 MappedSystemVa : 0xffffd00140134000 Void +0x020 StartVa : 0xffffd00140134000 Void
+0x028 ByteCount : 0
+0x02c ByteOffset : 0
+0x098 __Pfn : 0x22f52

It looks like the MDL the Packet structure itself is messed up.

xxxxx@gmail.com wrote:

Thanks David and Tim.

I do not have private symbols for xenvif,

Your dump suggests otherwise. The fact that it was able to pinpoint a
source file name and line number means that you DO have the symbols.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Tim, I got hold of the public symbols and the source. Thanks.

It seems like the Packet structure created by Xennet has a lot of empty fields. The Packet structure itself is generated from the Mdl.

FAULTING_IP:
xenvif!ReceiverReturnPacket+37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321]
fffff800`0b7926e3 498b1e mov rbx,qword ptr [r14]

DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT

BUGCHECK_STR: AV

PROCESS_NAME: System

ANALYSIS_VERSION: 6.3.9600.17237 (debuggers(dbg).140716-0327) amd64fre

DPC_STACK_BASE: FFFFF8006A1B2FB0

TRAP_FRAME: fffff8006a1b2570 – (.trap 0xfffff8006a1b2570)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=ffffe00113620300 rbx=0000000000000000 rcx=ffffe00113620300
rdx=ffffe001114f9da8 rsi=0000000000000000 rdi=0000000000000000
rip=fffff8000b7926e3 rsp=fffff8006a1b2700 rbp=fffff8006a1b2730
r8=0000000000000000 r9=0000000000000000 r10=ffffe001140c6880
r11=ffffe001113bb020 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
xenvif!__RingPutPacket [inlined in xenvif!ReceiverReturnPacket+0x37]:
fffff8000b7926e3 498b1e mov rbx,qword ptr [r14] ds:0000000000000000=???
Resetting default scope

LAST_CONTROL_TRANSFER: from fffff80066dd2ee9 to fffff80066dc73a0

STACK_TEXT:
fffff8006a1b2428 fffff80066dd2ee9 : 000000000000000a 0000000000000000 0000000000000002 0000000000000000 : nt!KeBugCheckEx
fffff8006a1b2430 fffff80066dd173a : 0000000000000000 0000000000000001 7473694c72656600 3030303030302073 : nt!KiBugCheckDispatch+0x69
fffff8006a1b2570 fffff8000b7926e3 : fffff800102f6250 ffffe001115ec010 0000000000000000 00000000ffffffff : nt!KiPageFault+0x23a
fffff8006a1b2700 fffff8000b79bef0 : ffffe001138dedf0 0000000000000001 ffffe0011386db98 ffffe0011386db80 : xenvif!ReceiverReturnPacket+0x37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321]
fffff8006a1b2750 fffff8000d568e95 : fffffffffffffffe ffffe0011367e280 fffff8006a1b2902 0000000000000000 : xenvif!VifReturnPacket+0x108 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\vif.c @ 385]
fffff8006a1b27a0 fffff8000a4770fd : ffffe001138dedf0 0000000000000000 0000000000000000 0000000000000003 : xennet!ReceiverReturnNetBufferLists+0x4d [c:\jenkins\jobs\beta-build-xennet\workspace\src\xennet\receiver.c @ 268]
fffff8006a1b27d0 fffff8000a475f60 : ffffe00110d83008 fffff8006a1b29c0 ffffe0011377f1a0 0000000000000000 : NDIS!ndisReturnNetBufferListsInternal+0x12d
fffff8006a1b2830 fffff8000a476094 : ffffe0011377f1a0 ffff11be65d24601 000000000000ce0e ffffe00110e6c880 : NDIS!ndisMTopReceiveNetBufferLists+0x30e
fffff8006a1b28c0 fffff8000d568dd2 : 0000000000000000 fffff8006a1b2b00 ffffe0011367e280 ffffd000f5478000 : NDIS!NdisMIndicateReceiveNetBufferLists+0x114
fffff8006a1b2ab0 fffff8000d568345 : 0000000000000000 0000000000000000 ffffe001138dedf0 ffffe001137cf010 : xennet!ReceiverReceivePackets+0x1e6 [c:\jenkins\jobs\beta-build-xennet\workspace\src\xennet\receiver.c @ 443]
fffff8006a1b2b20 fffff8000b79234f : ffffe00113620300 fffff80000000002 fffff8006a1b2b70 0000000000000000 : xennet!AdapterVifCallback+0x35 [c:\jenkins\jobs\beta-build-xennet\workspace\src\xennet\adapter.c @ 372]
fffff8006a1b2b50 fffff8000b78d8a0 : 0000000000000001 ffffe0011386d3d0 ffffe001113bb020 ffffe0011386d3f0 : xenvif!ReceiverNotify+0xa7 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3371]
fffff8006a1b2ba0 fffff80066ce3bf0 : fffff80066f78f00 fffff8006a1b2ce0 fffff8006a1b2e70 0000000000000000 : xenvif!NotifierDpc+0x74 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\notifier.c @ 159]
fffff8006a1b2be0 fffff80066ce2f37 : 0000000000000000 ffffe00113f03080 fffff80066f76180 fffff80000000001 : nt!KiExecuteAllDpcs+0x1b0
fffff8006a1b2d30 fffff80066dca4d5 : 0000000000000000 fffff80066f76180 0000000000000000 ffffe001140c6880 : nt!KiRetireDpcList+0xd7
fffff8006a1b2fb0 fffff80066dca2d9 : 0000000000000000 0000000000000000 ffffe001140c6880 0000000000000000 : nt!KxRetireDpcList+0x5
ffffd000f7c72900 fffff80066dcc3fa : ffffd000f7c729e0 fffff80066e5e4af ffff39bede7bddad 0000000000000001 : nt!KiDispatchInterruptContinue
ffffd000f7c72930 fffff80066cdd39e : 0000000000000000 fffff80066cf8296 0000000000000010 0000000000000246 : nt!KiDpcInterrupt+0xca
ffffd000f7c72ac0 fffff80066cf8656 : 0000000000000001 fffff80066f76180 0000000000000102 fffff80066cf80d5 : nt!KiExitThreadWait+0x32
ffffd000f7c72af0 fffff800102c9e84 : fffff800102f9bf0 ffffe00100000000 fffff800102f0500 0000000000000001 : nt!KeWaitForSingleObject+0x596
ffffd000f7c72b80 fffff80066d73f1c : fffff800102f9bf0 ffffe00110d56180 0000000000000000 0000000000000000 : ovsim!OvsStartIpHelper+0x244 [d:\ovs-windows\datapath-windows\ovsext\iphelper.c @ 1524]
ffffd000f7c72c00 fffff80066dcd9c6 : fffff80066f76180 ffffe001140c6880 ffffe00113f3b080 fffff80066cf6877 : nt!PspSystemThreadStartup+0x58
ffffd000f7c72c60 0000000000000000 : ffffd000f7c73000 ffffd000f7c6d000 0000000000000000 0000000000000000 : nt!KiStartSystemThread+0x16

STACK_COMMAND: kb

FOLLOWUP_IP:
xenvif!ReceiverReturnPacket+37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321]
fffff800`0b7926e3 498b1e mov rbx,qword ptr [r14]

FAULTING_SOURCE_LINE: c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c

FAULTING_SOURCE_FILE: c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c

FAULTING_SOURCE_LINE_NUMBER: 3321

SYMBOL_STACK_INDEX: 3

SYMBOL_NAME: xenvif!ReceiverReturnPacket+37

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: xenvif

IMAGE_NAME: xenvif.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 577b672b

BUCKET_ID_FUNC_OFFSET: 37

FAILURE_BUCKET_ID: AV_xenvif!ReceiverReturnPacket

BUCKET_ID: AV_xenvif!ReceiverReturnPacket

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:av_xenvif!receiverreturnpacket

FAILURE_ID_HASH: {0c093ea2-c479-d425-20c9-41e400fd1dca}

Followup: MachineOwner

0: kd> kn

Child-SP RetAddr Call Site

00 fffff8006a1b2428 fffff80066dd2ee9 nt!KeBugCheckEx
01 fffff8006a1b2430 fffff80066dd173a nt!KiBugCheckDispatch+0x69
02 fffff8006a1b2570 fffff8000b7926e3 nt!KiPageFault+0x23a
03 (Inline Function) ---------------- xenvif!__RingReturnPacket+0x13 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 1386] 04 fffff8006a1b2700 fffff8000b79bef0 xenvif!ReceiverReturnPacket+0x37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321] 05 fffff8006a1b2750 fffff8000d568e95 xenvif!VifReturnPacket+0x108 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\vif.c @ 385] 06 (Inline Function) ---------------- xennet!__ReceiverReturnNetBufferLists+0x3e [c:\jenkins\jobs\beta-build-xennet\workspace\src\xennet\receiver.c @ 248]
07 fffff8006a1b27a0 fffff8000a4770fd xennet!ReceiverReturnNetBufferLists+0x4d [c:\jenkins\jobs\beta-build-xennet\workspace\src\xennet\receiver.c @ 268]
08 fffff8006a1b27d0 fffff8000a475f60 NDIS!ndisReturnNetBufferListsInternal+0x12d
09 fffff8006a1b2830 fffff8000a476094 NDIS!ndisMTopReceiveNetBufferLists+0x30e
0a fffff8006a1b28c0 fffff8000d568dd2 NDIS!NdisMIndicateReceiveNetBufferLists+0x114
0b (Inline Function) ---------------- xennet!ReceiverPushPackets+0x72 [c:\jenkins\jobs\beta-build-xennet\workspace\src\xennet\receiver.c @ 373] 0c fffff8006a1b2ab0 fffff8000d568345 xennet!ReceiverReceivePackets+0x1e6 [c:\jenkins\jobs\beta-build-xennet\workspace\src\xennet\receiver.c @ 443] 0d fffff8006a1b2b20 fffff8000b79234f xennet!AdapterVifCallback+0x35 [c:\jenkins\jobs\beta-build-xennet\workspace\src\xennet\adapter.c @ 372] 0e (Inline Function) ---------------- xenvif!__RingReleaseLock+0x6a [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 1332]
0f (Inline Function) ---------------- xenvif!__RingNotify+0x7c [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 2669] 10 fffff8006a1b2b50 fffff8000b78d8a0 xenvif!ReceiverNotify+0xa7 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3371] 11 fffff8006a1b2ba0 fffff80066ce3bf0 xenvif!NotifierDpc+0x74 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\notifier.c @ 159] 12 fffff8006a1b2be0 fffff80066ce2f37 nt!KiExecuteAllDpcs+0x1b0 13 fffff8006a1b2d30 fffff80066dca4d5 nt!KiRetireDpcList+0xd7 14 fffff8006a1b2fb0 fffff80066dca2d9 nt!KxRetireDpcList+0x5 15 ffffd000f7c72900 fffff80066dcc3fa nt!KiDispatchInterruptContinue 16 ffffd000f7c72930 fffff80066cdd39e nt!KiDpcInterrupt+0xca 17 ffffd000f7c72ac0 fffff80066cf8656 nt!KiExitThreadWait+0x32 18 ffffd000f7c72af0 fffff800102c9e84 nt!KeWaitForSingleObject+0x596 19 ffffd000f7c72b80 fffff80066d73f1c ovsim!OvsStartIpHelper+0x244 [d:\ovs-windows\datapath-windows\ovsext\iphelper.c @ 1524] 1a ffffd000f7c72c00 fffff80066dcd9c6 nt!PspSystemThreadStartup+0x58 1b ffffd000f7c72c60 0000000000000000 nt!KiStartSystemThread+0x16 0: kd\> .frame /r 04 04 fffff8006a1b2700 fffff8000b79bef0 xenvif!ReceiverReturnPacket+0x37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321] rax=ffffe00113620300 rbx=0000000000000001 rcx=ffffe00113620300 rdx=ffffe001114f9da8 rsi=0000000000000000 rdi=ffffe001114f9e10 rip=fffff8000b7926e3 rsp=fffff8006a1b2700 rbp=fffff8006a1b2730 r8=0000000000000000 r9=0000000000000000 r10=ffffe001140c6880 r11=ffffe001113bb020 r12=0000000000000002 r13=0000000000000003 r14=0000000000000000 r15=ffffe00113620300 iopl=0 nv up ei ng nz na po nc cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00000286 xenvif!ReceiverReturnPacket+0x37: fffff8000b7926e3 498b1e mov rbx,qword ptr [r14] ds:002b:0000000000000000=???????????????? 0: kd\> dv Receiver = 0xffffe00113620300
Packet = 0xffffe001114f9da8 Ring = 0x0000000000000000
0: kd> u fffff8000b7926e3 xenvif!ReceiverReturnPacket+0x37 [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3321]: fffff8000b7926e3 498b1e mov rbx,qword ptr [r14]
fffff8000b7926e6 33d2 xor edx,edx fffff8000b7926e8 488d4f98 lea rcx,[rdi-68h]
fffff8000b7926ec 448d4268 lea r8d,[rdx+68h] fffff8000b7926f0 e8abbc0000 call xenvif!memset (fffff8000b79e3a0) fffff8000b7926f5 83672c00 and dword ptr [rdi+2Ch],0
fffff8000b7926f9 83672800 and dword ptr [rdi+28h],0 fffff8000b7926fd 488b4720 mov rax,qword ptr [rdi+20h]
0: kd> ub fffff8000b7926e3 xenvif!ReceiverReturnPacket+0x1b [c:\jenkins\jobs\beta-build-xenvif\workspace\src\xenvif\receiver.c @ 3318]: fffff8000b7926c7 4c8b7260 mov r14,qword ptr [rdx+60h]
fffff8000b7926cb 4883626000 and qword ptr [rdx+60h],0 fffff8000b7926d0 488d7a68 lea rdi,[rdx+68h]
fffff8000b7926d4 4c8bf9 mov r15,rcx fffff8000b7926d7 4885ff test rdi,rdi
fffff8000b7926da 744a je xenvif!ReceiverReturnPacket+0x7a (fffff8000b792726)
fffff8000b7926dc 488b37 mov rsi,qword ptr [rdi] fffff8000b7926df 48832700 and qword ptr [rdi],0
0: kd> dv /v
@r15 Receiver = 0xffffe00113620300 @rdx Packet = 0xffffe001114f9da8
@r14 Ring = 0x0000000000000000 0: kd\> dt Packet Local var @ rdx Type _XENVIF_RECEIVER_PACKET\* +0x000 ListEntry : _LIST_ENTRY [0x0000000000000000 - 0x00000000`00000000]
+0x010 Offset : 0
+0x014 Length : 0
+0x018 Info : _XENVIF_PACKET_INFO
+0x058 Flags : _XENVIF_CHECKSUM_FLAGS
+0x05c TagControlInformation : 0
+0x05e MaximumSegmentSize : 0
+0x060 Cookie : (null)
+0x068 Mdl : _MDL
+0x098 __Pfn : 0x103728

0: kd> dt -r Packet
Local var @ rdx Type _XENVIF_RECEIVER_PACKET*
+0x000 ListEntry : _LIST_ENTRY [0x0000000000000000 - 0x0000000000000000]
+0x000 Flink : (null)
+0x008 Blink : (null)
+0x010 Offset : 0
+0x014 Length : 0
+0x018 Info : _XENVIF_PACKET_INFO
+0x000 EthernetHeader : _XENVIF_PACKET_HEADER
+0x000 Offset : 0
+0x004 Length : 0
+0x008 LLCSnapHeader : _XENVIF_PACKET_HEADER
+0x000 Offset : 0
+0x004 Length : 0
+0x010 IpHeader : _XENVIF_PACKET_HEADER
+0x000 Offset : 0
+0x004 Length : 0
+0x018 IpOptions : _XENVIF_PACKET_HEADER
+0x000 Offset : 0
+0x004 Length : 0
+0x020 TcpHeader : _XENVIF_PACKET_HEADER
+0x000 Offset : 0
+0x004 Length : 0
+0x028 TcpOptions : _XENVIF_PACKET_HEADER
+0x000 Offset : 0
+0x004 Length : 0
+0x030 UdpHeader : _XENVIF_PACKET_HEADER
+0x000 Offset : 0
+0x004 Length : 0
+0x038 Flags : _XENVIF_PACKET_FLAGS
+0x000 IsAFragment : 0y0
+0x000 Reserved : 0y0000000000000000000000000000000 (0)
+0x03c Length : 0
+0x058 Flags : _XENVIF_CHECKSUM_FLAGS
+0x000 IpChecksumSucceeded : 0y0
+0x000 IpChecksumFailed : 0y0
+0x000 IpChecksumPresent : 0y0
+0x000 TcpChecksumSucceeded : 0y0
+0x000 TcpChecksumFailed : 0y0
+0x000 TcpChecksumPresent : 0y0
+0x000 UdpChecksumSucceeded : 0y0
+0x000 UdpChecksumFailed : 0y0
+0x000 UdpChecksumPresent : 0y0
+0x000 Reserved : 0y00000000000000000000000 (0)
+0x05c TagControlInformation : 0
+0x05e MaximumSegmentSize : 0
+0x060 Cookie : (null)
+0x068 Mdl : _MDL
+0x000 Next : (null)
+0x008 Size : 0n56
+0x00a MdlFlags : 0n3
+0x010 Process : (null)
+0x018 MappedSystemVa : 0xffffd000f54b7000 Void +0x020 StartVa : 0xffffd000f54b7000 Void
+0x028 ByteCount : 0
+0x02c ByteOffset : 0
+0x098 __Pfn : 0x103728

Based on my debugging, the Packet structure seems to be messed up for some reason. What is really weird is this happens only with Wireshark enabled. If I just set the packetFilter on the protocol edge to 0 to drop packet, this doesn’t seem to happen. So I am guessing this could be something to do with loopback packets.

How exactly should I debug this further? Why is the Receive code in the underlying driver crashing only when Wireshark is started? Sometimes, I have noticed I don’t even have to start capturing on an interface to see the crash, just opening up wireshark is enough for this.

I also do not see any issues using Netmon.

Netmon uses a LWF and does not modify the global packet filter settings.

Wireshark uses WinPCAP which is a Protocol. To see TX it must set the packet filter to All Local or Promiscuous. The side effect of that can be severe especially if other protocols or components don’t handle that correctly.

Of course the first place to look is if you component handles it correctly.

Good Luck,
Dave Cattley

Thanks David. Wireshark with my driver works with an Intel backed driver.
However, in a different environment where my NIC is backed by Xen, it
results in a crash in Xen. However, this happens only with my IM driver, so
it’s pretty difficult to figure out the exact cause.
On Nov 2, 2016 4:44 PM, “Dave Cattley” wrote:

> Netmon uses a LWF and does not modify the global packet filter settings.
>
>
>
> Wireshark uses WinPCAP which is a Protocol. To see TX it must set the
> packet filter to All Local or Promiscuous. The side effect of that can be
> severe especially if other protocols or components don’t handle that
> correctly.
>
>
>
> Of course the first place to look is if you component handles it correctly.
>
>
>
> Good Luck,
>
> Dave Cattley
>
>
>
>
>

In fact, I get a BSOD without even having WinPcap installed. Just opening Wireshark is enough to cause a crash.

That sounds like wireshark is sending some ioctl (oid) to the nic, and the nic chokes on the request.

Jan

On 11/2/16, 8:41 PM, “xxxxx@lists.osr.com on behalf of xxxxx@gmail.com” wrote:

In fact, I get a BSOD without even having WinPcap installed. Just opening Wireshark is enough to cause a crash.