I have a customer experiencing a memory leak with my NDIS6 driver. An SMB session from a network camera fails to connect, and also leaks some memory in tcpip.sys.
My drivers are PV drivers for Xen. Without the drivers loaded, Xen provides an emulated PCI device which works fine. With my drivers the PCI device is ‘unplugged’ and my drivers operate instead, so it is reasonably easy to determine that my drivers are a factor in the problem.
The customer has sent me a packet capture of the connection and simply by injecting the first packet using netcat on a Linux machine I’m able to reproduce the problem - poolmon tells me that a tag TcDN leaks 4624 bytes every time that data is received. It’s not just a transient allocation that is later freed either - by putting the packet injection in ain infinite loop the system easily chew through a few hundred MB wiithin a few minutes and poolmon shows no frees.
I’ve run wireshark in the VM and the data it captures (76 bytes of TCP data, 142 bytes of packet data in total) is identical to the data being sent, so I’m not corrupting anything anywhere at that level so I’m guessing it’s my handling of buffers that is the issue here.
Google tells me that TcDN is “TCP Delayed Delivery Network Buffer Lists”, my theory so far is that I’m mucking up the buffers that I’m giving to windows, which in turn upsets tcpip.sys causing it to not free a buffer it has allocated.
Any hints as to how to debug this issue? Will a checked version of tcpip.sys run on a free build of 2008R2?
Thanks
James
Does memory leak happen if you disable options in your driver, like LSO (TSO) or other offloads?
Is the problematic packet SYN with data?
>
Does memory leak happen if you disable options in your driver, like LSO
(TSO) or other offloads?
Hmmm… hadn’t thought of that. I’ll try that shortly. LSO isn’t invoked because the packets in question are tiny, but checksum offload will be being done.
Is the problematic packet SYN with data?
It goes like this:
-> SYN
<- SYN+ACK
-> ACK
-> SMB packet (76 bytes)
<- Negative session response, Unspecified error
It’s only that specific SMB packet that is causing the problem, so I’m guessing the handshake packets are okay or else it would have leaked all its memory long before this.
Thanks
James
Does the problem only occur when there is SMB failure, or SMB failure is part of the problem?
>Yes, and Yes.
Does that mean that the emulated driver doesn’t fail SMB connection?
Is it your driver that sends the SMB packer? Is packet contents the same as the emulated driver would send?
>
>Yes, and Yes.
Does that mean that the emulated driver doesn’t fail SMB connection?
That is correct.
Is it your driver that sends the SMB packer? Is packet contents the same as
the emulated driver would send?
The SMB packet is sent by a network video camera. I captured the SMB packet, dumped it to a file, then just sent it via netcat from another physical machine to the xen VM.
In the VM using wireshark I have determined that the packet seen by wireshark is identical whether my PV drivers or the PCI drivers for the emulated nic are used.
James
Do you have DriverVerifier enabled on the SUT?
>
Do you have DriverVerifier enabled on the SUT?
Yes. Verifying tcpip.sys, ndis.sys, xennet.sys (my network driver), and xenpci.sys (bus driver that enumerates PDO’s for xennet.sys and others)
Is there an easy way to get a checked build of tcpip.sys and ndis.sys without obtaining the entire checked install of Windows? And if I did obtain those, would they work in isolation without everything else being checked?
Thanks
James
What verifier settings are enabled (!verifier)?
Do you see the problem when verifier is disabled?
Poolmon shows the problem without the verifier running.
I’ll have to check the verifier setting later
Sent from my iPhone
On 23/12/2011, at 14:51, “xxxxx@broadcom.com” wrote:
> What verifier settings are enabled (!verifier)?
>
> Do you see the problem when verifier is disabled?
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
Checked build should be available for MSDN subscribers. Just install the whole checked build.
>
What verifier settings are enabled (!verifier)?
Settings are everying on except “Force pending I/O requests”, “Low resources simulation”, and “IRP Logging”
I’m now running on a checked build of 2008R2sp1 but it isn’t flagging any errors (apart from something I’m doing wrong on shutdown in the bus driver). Is there any more checking I can do in NDIS or tcpip to get it to check things more closely?
James
Keep in mind that Driver Verifier will not find EVERY possible logic
error. For example, if you fail to complete an IRP and return
STATUS_SUCCESS, you will get a warning that you have failed to pass it
down to another driver, failed to mark it pending and/or returned the
wrong status. But if you fix it to pend properly, and then never complete
it, Verifier won’t catch the fact that you failed to complete it and lost
the only pointer to the IRP.
Similarly, if some component allocates storage but does not free it, that
will not be detected until that component attempts to unload. So if you
force some level of NDIS to allocate on your behalf, fail to release it
yourself, and that NDIS component never unloads, Verifier won’t catch it.
That is, you can lie convincingly to Driver Verifier, and until you reach
a point where your lie can be checked and found to be untrue, Verifier
trusts you. This *might* be what is happening here. Verifier is just a
tool; a very effective one in some cases, but it can’t catch every lie up
front.
joe
>
> What verifier settings are enabled (!verifier)?
>
Settings are everying on except “Force pending I/O requests”, “Low
resources simulation”, and “IRP Logging”
I’m now running on a checked build of 2008R2sp1 but it isn’t flagging any
errors (apart from something I’m doing wrong on shutdown in the bus
driver). Is there any more checking I can do in NDIS or tcpip to get it to
check things more closely?
James
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
>
Similarly, if some component allocates storage but does not free it, that will
not be detected until that component attempts to unload. So if you force
some level of NDIS to allocate on your behalf, fail to release it yourself, and
that NDIS component never unloads, Verifier won’t catch it.
I can’t see that I’m forcing anything to allocate memory on my behalf. The memory that is leaking comes from tcpip.sys which I don’t talk to directly, and it’s over 4KB which is much bigger than the packets that trigger the problem.
But yes, the driver verifier, ndis verifier, and checked build were what I was trying first to save having to figure out the problem myself using the debugger. I knew there was no guarantee they’d pick up anything, especially since I’d already run all the WLK tests previously without any problems (which turns on the verifier too I think).
Fortunately I can pinpoint the problem to an exact packet so breaking on that packet and examining it in detail before I pass it to ndis should be easy… I’m just too lazy at this time of year 
Thanks
James
>
I have a customer experiencing a memory leak with my NDIS6 driver. An SMB
session from a network camera fails to connect, and also leaks some memory
in tcpip.sys.
My drivers are PV drivers for Xen. Without the drivers loaded, Xen provides
an emulated PCI device which works fine. With my drivers the PCI device is
‘unplugged’ and my drivers operate instead, so it is reasonably easy to
determine that my drivers are a factor in the problem.
The customer has sent me a packet capture of the connection and simply by
injecting the first packet using netcat on a Linux machine I’m able to
reproduce the problem - poolmon tells me that a tag TcDN leaks 4624 bytes
every time that data is received. It’s not just a transient allocation that is later
freed either - by putting the packet injection in ain infinite loop the system
easily chew through a few hundred MB wiithin a few minutes and poolmon
shows no frees.
I’ve run wireshark in the VM and the data it captures (76 bytes of TCP data,
142 bytes of packet data in total) is identical to the data being sent, so I’m not
corrupting anything anywhere at that level so I’m guessing it’s my handling of
buffers that is the issue here.
Google tells me that TcDN is “TCP Delayed Delivery Network Buffer Lists”, my
theory so far is that I’m mucking up the buffers that I’m giving to windows,
which in turn upsets tcpip.sys causing it to not free a buffer it has allocated.
Any hints as to how to debug this issue? Will a checked version of tcpip.sys
run on a free build of 2008R2?
After some further digging, I have found that the problem occurs on any connection to port 139 with a total packet size between 134 and the maximum (eg 1514 for a regular Ethernet network). The data in the packet doesn’t matter - I’m using random data. The thing that makes 134 a magic number is that windows sets my lookahead to 120 bytes giving a 134 byte value including the Ethernet header.
My drivers get the packet data from Linux/Xen which comes from Linux so the packets could consist of anywhere from 1 to 18 buffers. Windows seems to require that at least the packet header (eth+ip+tcp) are all inside the first MDL, so I consolidate everything up to current_lookahead into the first MDL even if Linux/Xen didn’t. Obviously that’s not enough for windows in this case though, but I can’t see what I’m doing wrong.
NDIS6.1 appears to have a thing called “Header-data split” that I’m not using, and the docs state for when you aren’t using it (http://msdn.microsoft.com/en-us/library/windows/hardware/ff544869(v=vs.85).aspx):
“All Ethernet frames that are not split must follow the general NDIS rules and requirements. For example, the first MDL in the chain of MDLs in a received NET_BUFFER structure must contain either the lookahead part of the frame or the entire Ethernet frame (whichever is smaller) in a virtually contiguous buffer. NDIS sets the size of lookahead with the OID_GEN_CURRENT_LOOKAHEAD OID.”
Help?
Thanks
James