RE: Ethernet Frame Type

OSR_Community_User · October 28, 2009, 2:15pm

I have another question regarding the location of the TCP Header in MDLs
I’m hoping somebody could answer, as I can’t seem to find it documented
anywhere …

I can find the Ethernet header doing the following:

ethHeader = (PEthHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf),
NormalPagePriority) + NET_BUFFER_DATA_OFFSET(NetBuf));

Based on Tom’s response below, if I encounter a Net Buffer that is
spanned across multiple MDLs (e.g. - EthHeader on 1st MDL, IP Header on
the 2nd MDL, and TCP Header on the 3rd MDL), I can walk the chain to
find the TCP header on the 3rd MDL.

Here is my code to get the TCP header if this is the case:

TCPHeader = (PTCPHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf)->Next->Next,
NormalPagePriority));

What I don’t know is should this statement really be:

TCPHeader = (PTCPHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf)->Next->Next,
NormalPagePriority) + NET_BUFFER_DATA_OFFSET(NetBuf));

Do I need to add the Net Buffer data offset if the TCP header is not on
the first MDL? So far my guess is no. I would verify this myself, but
I have never seen that portion of my code actually execute. So far,
it looks like everything is usually contained within 1 MDL, even though
I know it doesn’t have to be.

Thanks,

n Jim

From: Munafo, Jim (Xetron)
Sent: Thursday, August 13, 2009 2:34 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ethernet Frame Type

Thanks Tom,

Right now I get passed a NET_BUFFER_LIST. There is only 1 NET_BUFFER in
this list, and there is only 1 MDL in that NET_BUFFER. MappedSystemVa
in that MDL points to the beginning of the ARP header.

I think I just solved my problem though, and I’m probably just being an
idiot today. I’m looking at the following in a TCP/IP book:

ARP Packet Format:

|Ethernet Header| |ARP
Request/Reply|

[eth dest addr][eth src addr][type] [hard type][prot
type] […]

I just looked at the section name (ARP Packet Format) and assumed that
what I was looking at was the ARP packet format. However, it clearly
indicates that I’m looking at the Ethernet Header in the image of the
format in the book right before the ARP Request/Reply.

Don’t even know what to say. Thanks for the help though.

n Jim

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Thomas F. Divine
Sent: Thursday, August 13, 2009 2:07 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ethernet Frame Type

Jim,

You do understand that each NET_BUFFER carries a linked-list of MDLs -
not just a single MDL. You can’t make assumptions about the organization
of the singly-linked list of MDLs other than you should expect the
unexpected.

To access data in a NET_BUFFER you must write code that walks the MDL
chain to the desired offset. For example, the first MDL may be only 14
bytes long (ignoring backfill, etc…) and contain the Ethernet header.
The next MDL may contain the IP header, the next the TCP header, and
finally a MDL ( or more…) containing TCP data.

It’s a little tedious but just write code that “walks the chain” until
you finally reach a MDL that contains data at the offset of interest.

You can assume that headers are not split in the middle of a MDL.

Good luck,

Thomas F. Divine

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Munafo, Jim
(Xetron)
Sent: Thursday, August 13, 2009 1:39 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Ethernet Frame Type

I’m currently looking at a NET_BUFFER, and trying to determine the type
of frame that exist in the MDL. For example, I can get an ARP packet by
doing the following:

ULONG offset;

PMDL currentMDL;

PUCHAR packet;

offset = someNetBufferThatWasPassedToAFunction->DataOffset;

currentMDL = someNetBufferThatWasPassedToAFunction->CurrentMdl;

packet = (PUCHAR) currentMDL->MappedSystemVa;

packet += offset;

From here, packet points to the correct ARP packet, and I can look at
anything I need in the ARP packet. The problem I am having is how do I
know that this was an ARP packet and not something else (IP Packet). I
suppose I can look 12 bytes into the packet and see if bytes 13 & 14 are
0x0806 (indicating an ARP packet in the ARP packet format), but this is
ghetto.

What I really want is the location of the Ethernet encapsulated header.
That way, I could just look at the type field in the Ethernet frame and
determine the type of packet. However, I can’t find this in the MDL. I
looked at all of the bytes from the StartVa to the MappedSystemVa and
none of them look like the Ethernet frame.

Does anybody know where that is located or have a way of determining the
type of frame from a NET_BUFFER?

Thanks,

n Jim

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Igor_Sharovar · October 28, 2009, 4:02pm

>Do I need to add the Net Buffer data offset if the TCP header is not on the >first MDL? So far my guess is no. I would verify this myself, but I have never >seen that portion of my code actually execute. So far, it looks like >everything is usually contained within 1 MDL, even though I know it doesn’t >have to be.
You should expect both situations. You could have just one MDL which contains all data including TCP payload and a situation when you will have more than one MDL - each MDL for Ethernet header, IP header and TCP payload. It would depends on a NIC capability.
If you get just one MDL just do parsing Ethernet Header->Layer 3 header -> Layer 4 header( if available). If your NET_BUFFER_DATA->Next is not NULL check what kind of Layer 3 protocol you have. You could have not only IP packets but others Layer 3 protocols. For example it could be ARP. If it is IP you could check ->Next field again to retrieve TCP data. But probably it is better to use NET_BUFFER macro like NET_BUFFER_NEXT_NB.
The bottom line is that you should always check the next MDL first and not doing like this

MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf)->>Next->Next, NormalPagePriority) + NET_BUFFER_DATA_OFFSET(NetBuf));

Igor Sharovar

OSR_Community_User · October 28, 2009, 4:39pm

Thanks Igor. I didn’t actually post all of my code, as I do check the
next MDL first as you suggest. I just don’t know if that line should
actually contain the “+ NET_BUFFER_DATA_OFFSET(NetBuf));” part since it
would not be on the first MDL. Do I need to add the
NET_BUFFER_DATA_OFFSET to the pointer if it is not on the first MDL?

Thanks again.
– Jim

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@hotmail.com
Sent: Wednesday, October 28, 2009 4:01 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] RE: Ethernet Frame Type

Do I need to add the Net Buffer data offset if the TCP header is not on
the >first MDL? So far my guess is no. I would verify this myself, but I
have never >seen that portion of my code actually execute. So far,
it looks like >everything is usually contained within 1 MDL, even though
I know it doesn’t >have to be.
You should expect both situations. You could have just one MDL which
contains all data including TCP payload and a situation when you will
have more than one MDL - each MDL for Ethernet header, IP header and TCP
payload. It would depends on a NIC capability.
If you get just one MDL just do parsing Ethernet Header->Layer 3 header
-> Layer 4 header( if available). If your NET_BUFFER_DATA->Next is not
NULL check what kind of Layer 3 protocol you have. You could have not
only IP packets but others Layer 3 protocols. For example it could be
ARP. If it is IP you could check ->Next field again to retrieve TCP
data. But probably it is better to use NET_BUFFER macro like
NET_BUFFER_NEXT_NB.
The bottom line is that you should always check the next MDL first and
not doing like this

MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf)->>Next->Next
, NormalPagePriority) + NET_BUFFER_DATA_OFFSET(NetBuf));

Igor Sharovar

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Igor_Sharovar · October 28, 2009, 6:15pm

>Do I need to add the NET_BUFFER_DATA_OFFSET to the pointer if it is >not on the first MDL?
Jim,

I confused you. NET_BUFFER_DATA->Next contains a pointer to the next NET_BUFFER but not to MDL. You need to walk through MDL list of NET_BUFFER.
NET_BUFFER_DATA_OFFSET is used only to identify offset of data in one NET_BUFFER. As soon you get the first MDL entry by using NET_BUFFER_DATA_OFFSET you don’t need to call NET_BUFFER_DATA_OFFSET again. The next MDL element ( if any) will contain the next portion of data.

Igor Sharovar

David_R_Cattley · October 29, 2009, 1:38am

It is not surprising to find confusion about where the data is in a
NET_BUFFER. When explaining I find that working up from some basics makes
it easier:

Consider an MDL. It describes a virtually contiguous memory region. It
is a list of page frame numbers. The virtually contiguous region starts
somewhere in the first page. The ‘distance’ into the first page is returned
by MmGetMdlByteOffset(). For the most part you will simply ignore this and
get the system virtual address of the first byte of the memory region by
calling MmGetSystemAddressForMdlSafe(). The total length of the virtually
contiguous memory region is returned by MmGetMdlByteCount().

Now consider a chain of MDLs linked together. This chain (MDL chain or
NDIS_BUFFER chain) describe a ‘logically’ contiguous buffer that is
virtually discontiguous. It is made of pieces. Each piece is described
by an MDL. That means each piece starts at a virtual address and has a
length. The total length of the logical buffer is the sum of the lengths
of the buffer pieces. An item contained in the buffer may fit entirely
within a single piece and thus be virtually contiguous or it may straddle
one or more pieces and thus need to be accessed in multiple reads or writes
(or copied to/from a linear buffer for processing).

The story ended here with NDIS5. The MDL chain was the packet NDIS_BUFFER
chain. Enter NDIS6 and support for shrink/expand (or advance/retard in
NDIS6 terminology) of ‘buffers’.

An NDIS6 NET_BUFFER has an MDL chain. This chain describes a logically
contiguous but virtual discontiguous ‘buffer’. Somewhere in this buffer is
the data. The NET_BUFFER, however, can have additional space preceeding
and following the ‘data’. So in effect, you have ‘head’ and ‘tail’ room.
So a NET_BUFFER needs a way to describe where the data starts (in the
buffer) and where it ends.

Where it ends is easy - the NET_BUFFER contains a length. The buffer may be
longer than the length (but never shorter). The contents beyond the length
but before the ‘end of the logical buffer’ are effectively not part of the
packet described.

Where the beginning is is a bit trickier. A simple way to have described
this would have been an offset from the start of the logical buffer. Well
that is exactly what NET_BUFFER_DATA_OFFSET() is. If you start with the
logical buffer described by the MDL Chain from NET_BUFFER_FIRST_MDL() and
‘skip’ NET_BUFFER_DATA_OFFSET() bytes (dealing with the possible
discontinuity as you skip over MDLs) you land at the beginning of the data.

But that is really inefficient if the header space is long and possibly
described by multiple pieces (MDLs). Rolling through the MDL chain just to
skip the header space is dorky. So enter NET_BUFFER_CURRENT_MDL() and
NET_BUFFER_CURRENT_MDL_OFFSET(). These are ‘accelerators’ to the first MDL
in the MDL chain that actually has some data in it - IE, the start of the
data in the logical buffer. The NET_BUFFER keeps track of this point in the
MDL chain precisely to make it more convenient (and more efficient) to get
to the data.

The data may still span multiple MDLs of course.

I hope that helps. So for an Ethernet packet, the MAC header starts at

(PUCHAR)MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL()) +
NET_BUFFER_CURRENT_MDL_OFFSET()

If you ‘advance’ the NET_BUFFER the size of the MAC header then the IP
header will now be at the (new)

(PUCHAR)MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL()) +
NET_BUFFER_CURRENT_MDL_OFFSET()

Now the IP header is supposed to be contained in a virtually contiguous
region (IE, one MDL’s worth of buffer-piece). I don’t code drivers that
rely on that (or assume it blatantly) and so my drivers tend to have code
that will access the IP header directly *if* it is contiguous or bounce it
to a flat buffer (and access it in the flat buffer) if it is discontiguous.
Your mileage may vary.

Good Luck,
Dave Cattley

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Munafo, Jim (Xetron)
Sent: Wednesday, October 28, 2009 2:13 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ethernet Frame Type

I have another question regarding the location of the TCP Header in MDLs I?m
hoping somebody could answer, as I can?t seem to find it documented anywhere
?

I can find the Ethernet header doing the following:

ethHeader = (PEthHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf),
NormalPagePriority) + NET_BUFFER_DATA_OFFSET(NetBuf));

Based on Tom?s response below, if I encounter a Net Buffer that is spanned
across multiple MDLs (e.g. - EthHeader on 1st MDL, IP Header on the 2nd MDL,
and TCP Header on the 3rd MDL), I can walk the chain to find the TCP header
on the 3rd MDL.

Here is my code to get the TCP header if this is the case:
TCPHeader = (PTCPHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf)->Next->Next,
NormalPagePriority));

What I don?t know is should this statement really be:
TCPHeader = (PTCPHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf)->Next->Next,
NormalPagePriority) + NET_BUFFER_DATA_OFFSET(NetBuf));

Do I need to add the Net Buffer data offset if the TCP header is not on the
first MDL?? So far my guess is no.? I would verify this myself, but I have
never seen that portion of my code actually execute.? So far, it looks
like everything is usually contained within 1 MDL, even though I know it
doesn?t have to be.

Thanks,
* Jim

From: Munafo, Jim (Xetron)
Sent: Thursday, August 13, 2009 2:34 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ethernet Frame Type

Thanks Tom,

Right now I get passed a NET_BUFFER_LIST.? There is only 1 NET_BUFFER in
this list, and there is only 1 MDL in that NET_BUFFER.? MappedSystemVa in
that MDL points to the beginning of the ARP header.

I think I just solved my problem though, and I?m probably just being an
idiot today.? I?m looking at the following in a TCP/IP book:

ARP Packet Format:
|Ethernet Header|??? ???|ARP
Request/Reply|
[eth dest addr][eth src addr][type]??? [hard type][prot type]
[?]

I just looked at the section name (ARP Packet Format) and assumed that what
I was looking at was the ARP packet format.? However, it clearly indicates
that I?m looking at the Ethernet Header in the image of the format in the
book right before the ARP Request/Reply.

Don?t even know what to say.? Thanks for the help though.
* Jim

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Thomas F. Divine
Sent: Thursday, August 13, 2009 2:07 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ethernet Frame Type

Jim,

You do understand that each NET_BUFFER carries a linked-list of MDLs ? not
just a single MDL. You can?t make assumptions about the organization of the
singly-linked list of MDLs other than you should expect the unexpected.

To access data in a NET_BUFFER you must write code that walks the MDL chain
to the desired offset. For example, the first MDL may be only 14 bytes long
(ignoring backfill, etc?) and contain the Ethernet header. The next MDL may
contain the IP header, the next the TCP header, and finally a MDL ( or
more…) containing TCP data.

It?s a little tedious but just write code that ?walks the chain? until you
finally reach a MDL that contains data at the offset of interest.

You can assume that headers are not split in the middle of a MDL.

Good luck,

Thomas F. Divine

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Munafo, Jim (Xetron)
Sent: Thursday, August 13, 2009 1:39 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Ethernet Frame Type

I?m currently looking at a NET_BUFFER, and trying to determine the type of
frame that exist in the MDL.? For example, I can get an ARP packet by doing
the following:
ULONG offset;
PMDL currentMDL;
PUCHAR packet;
offset = someNetBufferThatWasPassedToAFunction->DataOffset;
currentMDL = someNetBufferThatWasPassedToAFunction->CurrentMdl;
packet = (PUCHAR) currentMDL->MappedSystemVa;
packet += offset;
From here, packet points to the correct ARP packet, and I can look at
anything I need in the ARP packet.? The problem I am having is how do I know
that this was an ARP packet and not something else (IP Packet).? I suppose I
can look 12 bytes into the packet and see if bytes 13 & 14 are 0x0806
(indicating an ARP packet in the ARP packet format), but this is ghetto.
What I really want is the location of the Ethernet encapsulated header.?
That way, I could just look at the type field in the Ethernet frame and
determine the type of packet.? However, I can?t find this in the MDL.? I
looked at all of the bytes from the StartVa to the MappedSystemVa and none
of them look like the Ethernet frame.
Does anybody know where that is located or have a way of determining the
type of frame from a NET_BUFFER?
Thanks,
T??? Jim

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

OSR_Community_User · October 29, 2009, 10:28am

Thanks guys for the response. Dave, that was an excellent explanation. That’s basically what I was thinking, but let me know if I’m still confused …

Right now, I get the Ethernet header via:
ethHeader = (PEthHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf),
NormalPagePriority) + NET_BUFFER_DATA_OFFSET(NetBuf));

This part seems fairly easy, and essentially does the same thing you suggest below:
(PUCHAR)MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL()) +
NET_BUFFER_CURRENT_MDL_OFFSET()

Though like you said, I’m sure your way is better due to the acceleration.

Anyway, this is where I may still be a little confused …
Suppose I have an ARP header on the next MDL. If I “advance” to the next MDL to get the ARP header, I do some checking with the current (first) MDL size to make sure that the ARP header is on the next MDL, and then I access it via the following:

arpHeader = (parpHeader) MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(netBuf)->Next, NormalPagePriority);

To me, that line of code above is saying, based on current Net Buffer, give me the current (first) MDL, and then point to the next MDL (second) in the chain (NET_BUFFER_CURRENT_MDL(netBuf)->Next [Points to the 2nd MDL]). This is how I’m “advancing”.

Based on your explanation, it seems as though that line of code is correct, and it would NOT be:

arpHeader = (parpHeader) ((PUCHAR) MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(netBuf)->Next, NormalPagePriority) + NET_BUFFER_DATA_OFFSET(NetBuf));

This is because:
This chain describes a logically contiguous but virtual discontiguous ‘buffer’. Like you said below. So I would only need to add the offset to the first NetBuf MDL to get to the start of the data. If that’s not correct, please let me know 8-).

Also, if there is another way to “advance”, than just using the MDL->Next pointer, that would be great to know as well. However, my driver has to work on XP too (I’ve got some old NDIS functionality in it) and I can’t use some of the latest NDIS APIs in my driver because it will fail to load on XP.

I really wish I could just get this code to execute so that I could test it to make sure that it works.

Igor,

Thanks for your responses as well. The last one:

Do I need to add the NET_BUFFER_DATA_OFFSET to the pointer if it is >not on the first MDL?
Jim,

I confused you. NET_BUFFER_DATA->Next contains a pointer to the next NET_BUFFER but not to MDL. You need to walk through MDL list of NET_BUFFER.
NET_BUFFER_DATA_OFFSET is used only to identify offset of data in one NET_BUFFER. As soon you get the first MDL entry by using NET_BUFFER_DATA_OFFSET you don’t need to call NET_BUFFER_DATA_OFFSET again. The next MDL element ( if any) will contain the next portion of data.

Igor Sharovar

I was probably a little unclear in my explanation, but I think the code:
TCPHeader = (PTCPHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf)->Next->Next,
NormalPagePriority));

Is really saying get the address of the first MDL [NET_BUFFER_CURRENT_MDL(NetBuf)] and then “advance” to the third MDL [NET_BUFFER_CURRENT_MDL(NetBuf)->Next->Next], not the next NET_BUFFER.

Thanks again,
– Jim

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of David R. Cattley
Sent: Wednesday, October 28, 2009 11:51 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ethernet Frame Type

It is not surprising to find confusion about where the data is in a
NET_BUFFER. When explaining I find that working up from some basics makes
it easier:

Consider an MDL. It describes a virtually contiguous memory region. It
is a list of page frame numbers. The virtually contiguous region starts
somewhere in the first page. The ‘distance’ into the first page is returned
by MmGetMdlByteOffset(). For the most part you will simply ignore this and
get the system virtual address of the first byte of the memory region by
calling MmGetSystemAddressForMdlSafe(). The total length of the virtually
contiguous memory region is returned by MmGetMdlByteCount().

Now consider a chain of MDLs linked together. This chain (MDL chain or
NDIS_BUFFER chain) describe a ‘logically’ contiguous buffer that is
virtually discontiguous. It is made of pieces. Each piece is described
by an MDL. That means each piece starts at a virtual address and has a
length. The total length of the logical buffer is the sum of the lengths
of the buffer pieces. An item contained in the buffer may fit entirely
within a single piece and thus be virtually contiguous or it may straddle
one or more pieces and thus need to be accessed in multiple reads or writes
(or copied to/from a linear buffer for processing).

The story ended here with NDIS5. The MDL chain was the packet NDIS_BUFFER
chain. Enter NDIS6 and support for shrink/expand (or advance/retard in
NDIS6 terminology) of ‘buffers’.

An NDIS6 NET_BUFFER has an MDL chain. This chain describes a logically
contiguous but virtual discontiguous ‘buffer’. Somewhere in this buffer is
the data. The NET_BUFFER, however, can have additional space preceeding
and following the ‘data’. So in effect, you have ‘head’ and ‘tail’ room.
So a NET_BUFFER needs a way to describe where the data starts (in the
buffer) and where it ends.

Where it ends is easy - the NET_BUFFER contains a length. The buffer may be
longer than the length (but never shorter). The contents beyond the length
but before the ‘end of the logical buffer’ are effectively not part of the
packet described.

Where the beginning is is a bit trickier. A simple way to have described
this would have been an offset from the start of the logical buffer. Well
that is exactly what NET_BUFFER_DATA_OFFSET() is. If you start with the
logical buffer described by the MDL Chain from NET_BUFFER_FIRST_MDL() and
‘skip’ NET_BUFFER_DATA_OFFSET() bytes (dealing with the possible
discontinuity as you skip over MDLs) you land at the beginning of the data.

But that is really inefficient if the header space is long and possibly
described by multiple pieces (MDLs). Rolling through the MDL chain just to
skip the header space is dorky. So enter NET_BUFFER_CURRENT_MDL() and
NET_BUFFER_CURRENT_MDL_OFFSET(). These are ‘accelerators’ to the first MDL
in the MDL chain that actually has some data in it - IE, the start of the
data in the logical buffer. The NET_BUFFER keeps track of this point in the
MDL chain precisely to make it more convenient (and more efficient) to get
to the data.

The data may still span multiple MDLs of course.

I hope that helps. So for an Ethernet packet, the MAC header starts at

(PUCHAR)MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL()) +
NET_BUFFER_CURRENT_MDL_OFFSET()

If you ‘advance’ the NET_BUFFER the size of the MAC header then the IP
header will now be at the (new)

(PUCHAR)MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL()) +
NET_BUFFER_CURRENT_MDL_OFFSET()

Now the IP header is supposed to be contained in a virtually contiguous
region (IE, one MDL’s worth of buffer-piece). I don’t code drivers that
rely on that (or assume it blatantly) and so my drivers tend to have code
that will access the IP header directly *if* it is contiguous or bounce it
to a flat buffer (and access it in the flat buffer) if it is discontiguous.
Your mileage may vary.

Good Luck,
Dave Cattley

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Munafo, Jim (Xetron)
Sent: Wednesday, October 28, 2009 2:13 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ethernet Frame Type

I have another question regarding the location of the TCP Header in MDLs I’m
hoping somebody could answer, as I can’t seem to find it documented anywhere
…

I can find the Ethernet header doing the following:

ethHeader = (PEthHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf),
NormalPagePriority) + NET_BUFFER_DATA_OFFSET(NetBuf));

Based on Tom’s response below, if I encounter a Net Buffer that is spanned
across multiple MDLs (e.g. - EthHeader on 1st MDL, IP Header on the 2nd MDL,
and TCP Header on the 3rd MDL), I can walk the chain to find the TCP header
on the 3rd MDL.

Here is my code to get the TCP header if this is the case:
TCPHeader = (PTCPHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf)->Next->Next,
NormalPagePriority));

What I don’t know is should this statement really be:
TCPHeader = (PTCPHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf)->Next->Next,
NormalPagePriority) + NET_BUFFER_DATA_OFFSET(NetBuf));

Do I need to add the Net Buffer data offset if the TCP header is not on the
first MDL?? So far my guess is no.? I would verify this myself, but I have
never seen that portion of my code actually execute.? So far, it looks
like everything is usually contained within 1 MDL, even though I know it
doesn’t have to be.

Thanks,
* Jim

From: Munafo, Jim (Xetron)
Sent: Thursday, August 13, 2009 2:34 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ethernet Frame Type

Thanks Tom,

Right now I get passed a NET_BUFFER_LIST.? There is only 1 NET_BUFFER in
this list, and there is only 1 MDL in that NET_BUFFER.? MappedSystemVa in
that MDL points to the beginning of the ARP header.

I think I just solved my problem though, and I’m probably just being an
idiot today.? I’m looking at the following in a TCP/IP book:

ARP Packet Format:
|Ethernet Header|??? ???|ARP
Request/Reply|
[eth dest addr][eth src addr][type]??? [hard type][prot type]
[…]

I just looked at the section name (ARP Packet Format) and assumed that what
I was looking at was the ARP packet format.? However, it clearly indicates
that I’m looking at the Ethernet Header in the image of the format in the
book right before the ARP Request/Reply.

Don’t even know what to say.? Thanks for the help though.
* Jim

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Thomas F. Divine
Sent: Thursday, August 13, 2009 2:07 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ethernet Frame Type

Jim,

You do understand that each NET_BUFFER carries a linked-list of MDLs - not
just a single MDL. You can’t make assumptions about the organization of the
singly-linked list of MDLs other than you should expect the unexpected.

To access data in a NET_BUFFER you must write code that walks the MDL chain
to the desired offset. For example, the first MDL may be only 14 bytes long
(ignoring backfill, etc…) and contain the Ethernet header. The next MDL may
contain the IP header, the next the TCP header, and finally a MDL ( or
more…) containing TCP data.

It’s a little tedious but just write code that “walks the chain” until you
finally reach a MDL that contains data at the offset of interest.

You can assume that headers are not split in the middle of a MDL.

Good luck,

Thomas F. Divine

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Munafo, Jim (Xetron)
Sent: Thursday, August 13, 2009 1:39 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Ethernet Frame Type

I’m currently looking at a NET_BUFFER, and trying to determine the type of
frame that exist in the MDL.? For example, I can get an ARP packet by doing
the following:
ULONG offset;
PMDL currentMDL;
PUCHAR packet;
offset = someNetBufferThatWasPassedToAFunction->DataOffset;
currentMDL = someNetBufferThatWasPassedToAFunction->CurrentMdl;
packet = (PUCHAR) currentMDL->MappedSystemVa;
packet += offset;
From here, packet points to the correct ARP packet, and I can look at
anything I need in the ARP packet.? The problem I am having is how do I know
that this was an ARP packet and not something else (IP Packet).? I suppose I
can look 12 bytes into the packet and see if bytes 13 & 14 are 0x0806
(indicating an ARP packet in the ARP packet format), but this is ghetto.
What I really want is the location of the Ethernet encapsulated header.?
That way, I could just look at the type field in the Ethernet frame and
determine the type of packet.? However, I can’t find this in the MDL.? I
looked at all of the bytes from the StartVa to the MappedSystemVa and none
of them look like the Ethernet frame.
Does anybody know where that is located or have a way of determining the
type of frame from a NET_BUFFER?
Thanks,
T??? Jim

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

David_R_Cattley · October 29, 2009, 11:16am

Actually, those are *not* equivalent.

The expression I gave you takes into account the fact that MDL that contains
the first part of the virtual buffer might actually have some space at the
beginning. Your expression starts at beginning of the segment of the
virtual buffer described by the current MDL. Those are subtly *not* the
same location.

Next point: “Advance” is an operation on the NET_BUFFER, not on an MDL. If
you use Advance and Retreat you are narrowing and widening the current
‘window’ on the logical buffer. This window always starts at
NET_BUFFER_CURRENT_MDL()+NET_BUFFER_CURRENT_MDL_OFFSET().

Keep in mind that behind the scenes of Xxx{Advance|Retreat} NDIS makes
adjustments to the value(s) returned by NET_BUFFER_CURRENT_MDL() and
NET_BUFFER_CURRENT_MDL_OFFSET() as necessary.

“Advance” is *NOT* the act of moving to the next MDL in an MDL chain.
Frankly, I don’t have a name for that but ‘traverse’ comes to mind.
“Advance” means something very specific to the current window on the logical
buffer.

So the ARP header start logically after the MAC header. That might be in
the same memory described by the MDL that describes where the MAC header is,
it might straddle that MDL’s memory and into the next, it might start in the
next MDL, etc. You have to deal with all three cases.

And now your third point: XP? You don’t have NET_BUFFERs in XP. You just
have NDIS_BUFFER chains (MDL chains) and the logically contiguous buffer
describes the packet. No head space, no tail space.

Good Luck,
Dave Cattley

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Munafo, Jim (Xetron)
Sent: Thursday, October 29, 2009 10:28 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ethernet Frame Type

Thanks guys for the response. Dave, that was an excellent explanation.
That’s basically what I was thinking, but let me know if I’m still confused
…

Right now, I get the Ethernet header via:
ethHeader = (PEthHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf),
NormalPagePriority) + NET_BUFFER_DATA_OFFSET(NetBuf));

This part seems fairly easy, and essentially does the same thing you suggest
below:
(PUCHAR)MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL()) +
NET_BUFFER_CURRENT_MDL_OFFSET()

Though like you said, I’m sure your way is better due to the acceleration.

Anyway, this is where I may still be a little confused …
Suppose I have an ARP header on the next MDL. If I “advance” to the next
MDL to get the ARP header, I do some checking with the current (first) MDL
size to make sure that the ARP header is on the next MDL, and then I access
it via the following:

arpHeader = (parpHeader)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(netBuf)->Next,
NormalPagePriority);

To me, that line of code above is saying, based on current Net Buffer, give
me the current (first) MDL, and then point to the next MDL (second) in the
chain (NET_BUFFER_CURRENT_MDL(netBuf)->Next [Points to the 2nd MDL]). This
is how I’m “advancing”.

Based on your explanation, it seems as though that line of code is correct,
and it would NOT be:

arpHeader = (parpHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(netBuf)->Next,
NormalPagePriority) + NET_BUFFER_DATA_OFFSET(NetBuf));

This is because:
This chain describes a logically contiguous but virtual discontiguous
‘buffer’. Like you said below. So I would only need to add the offset to
the first NetBuf MDL to get to the start of the data. If that’s not
correct, please let me know 8-).

Also, if there is another way to “advance”, than just using the MDL->Next
pointer, that would be great to know as well. However, my driver has to
work on XP too (I’ve got some old NDIS functionality in it) and I can’t use
some of the latest NDIS APIs in my driver because it will fail to load on
XP.

I really wish I could just get this code to execute so that I could test it
to make sure that it works.

Igor,

Thanks for your responses as well. The last one:

Do I need to add the NET_BUFFER_DATA_OFFSET to the pointer if it is >not on
the first MDL?
Jim,

I confused you. NET_BUFFER_DATA->Next contains a pointer to the next
NET_BUFFER but not to MDL. You need to walk through MDL list of NET_BUFFER.
NET_BUFFER_DATA_OFFSET is used only to identify offset of data in one
NET_BUFFER. As soon you get the first MDL entry by using
NET_BUFFER_DATA_OFFSET you don’t need to call NET_BUFFER_DATA_OFFSET again.
The next MDL element ( if any) will contain the next portion of data.

Igor Sharovar

I was probably a little unclear in my explanation, but I think the code:
TCPHeader = (PTCPHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf)->Next->Next,
NormalPagePriority));

Is really saying get the address of the first MDL
[NET_BUFFER_CURRENT_MDL(NetBuf)] and then “advance” to the third MDL
[NET_BUFFER_CURRENT_MDL(NetBuf)->Next->Next], not the next NET_BUFFER.

Thanks again,
– Jim

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of David R. Cattley
Sent: Wednesday, October 28, 2009 11:51 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ethernet Frame Type

It is not surprising to find confusion about where the data is in a
NET_BUFFER. When explaining I find that working up from some basics makes
it easier:

Consider an MDL. It describes a virtually contiguous memory region. It
is a list of page frame numbers. The virtually contiguous region starts
somewhere in the first page. The ‘distance’ into the first page is returned
by MmGetMdlByteOffset(). For the most part you will simply ignore this and
get the system virtual address of the first byte of the memory region by
calling MmGetSystemAddressForMdlSafe(). The total length of the virtually
contiguous memory region is returned by MmGetMdlByteCount().

Now consider a chain of MDLs linked together. This chain (MDL chain or
NDIS_BUFFER chain) describe a ‘logically’ contiguous buffer that is
virtually discontiguous. It is made of pieces. Each piece is described
by an MDL. That means each piece starts at a virtual address and has a
length. The total length of the logical buffer is the sum of the lengths
of the buffer pieces. An item contained in the buffer may fit entirely
within a single piece and thus be virtually contiguous or it may straddle
one or more pieces and thus need to be accessed in multiple reads or writes
(or copied to/from a linear buffer for processing).

The story ended here with NDIS5. The MDL chain was the packet NDIS_BUFFER
chain. Enter NDIS6 and support for shrink/expand (or advance/retard in
NDIS6 terminology) of ‘buffers’.

An NDIS6 NET_BUFFER has an MDL chain. This chain describes a logically
contiguous but virtual discontiguous ‘buffer’. Somewhere in this buffer is
the data. The NET_BUFFER, however, can have additional space preceeding
and following the ‘data’. So in effect, you have ‘head’ and ‘tail’ room.
So a NET_BUFFER needs a way to describe where the data starts (in the
buffer) and where it ends.

Where it ends is easy - the NET_BUFFER contains a length. The buffer may be
longer than the length (but never shorter). The contents beyond the length
but before the ‘end of the logical buffer’ are effectively not part of the
packet described.

Where the beginning is is a bit trickier. A simple way to have described
this would have been an offset from the start of the logical buffer. Well
that is exactly what NET_BUFFER_DATA_OFFSET() is. If you start with the
logical buffer described by the MDL Chain from NET_BUFFER_FIRST_MDL() and
‘skip’ NET_BUFFER_DATA_OFFSET() bytes (dealing with the possible
discontinuity as you skip over MDLs) you land at the beginning of the data.

But that is really inefficient if the header space is long and possibly
described by multiple pieces (MDLs). Rolling through the MDL chain just to
skip the header space is dorky. So enter NET_BUFFER_CURRENT_MDL() and
NET_BUFFER_CURRENT_MDL_OFFSET(). These are ‘accelerators’ to the first MDL
in the MDL chain that actually has some data in it - IE, the start of the
data in the logical buffer. The NET_BUFFER keeps track of this point in the
MDL chain precisely to make it more convenient (and more efficient) to get
to the data.

The data may still span multiple MDLs of course.

I hope that helps. So for an Ethernet packet, the MAC header starts at

(PUCHAR)MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL()) +
NET_BUFFER_CURRENT_MDL_OFFSET()

If you ‘advance’ the NET_BUFFER the size of the MAC header then the IP
header will now be at the (new)

(PUCHAR)MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL()) +
NET_BUFFER_CURRENT_MDL_OFFSET()

Now the IP header is supposed to be contained in a virtually contiguous
region (IE, one MDL’s worth of buffer-piece). I don’t code drivers that
rely on that (or assume it blatantly) and so my drivers tend to have code
that will access the IP header directly *if* it is contiguous or bounce it
to a flat buffer (and access it in the flat buffer) if it is discontiguous.
Your mileage may vary.

Good Luck,
Dave Cattley

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Munafo, Jim (Xetron)
Sent: Wednesday, October 28, 2009 2:13 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ethernet Frame Type

I have another question regarding the location of the TCP Header in MDLs I’m
hoping somebody could answer, as I can’t seem to find it documented anywhere
…

I can find the Ethernet header doing the following:

ethHeader = (PEthHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf),
NormalPagePriority) + NET_BUFFER_DATA_OFFSET(NetBuf));

Based on Tom’s response below, if I encounter a Net Buffer that is spanned
across multiple MDLs (e.g. - EthHeader on 1st MDL, IP Header on the 2nd MDL,
and TCP Header on the 3rd MDL), I can walk the chain to find the TCP header
on the 3rd MDL.

Here is my code to get the TCP header if this is the case:
TCPHeader = (PTCPHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf)->Next->Next,
NormalPagePriority));

What I don’t know is should this statement really be:
TCPHeader = (PTCPHeader) ((PUCHAR)
MmGetSystemAddressForMdlSafe(NET_BUFFER_CURRENT_MDL(NetBuf)->Next->Next,
NormalPagePriority) + NET_BUFFER_DATA_OFFSET(NetBuf));

Do I need to add the Net Buffer data offset if the TCP header is not on the
first MDL?? So far my guess is no.? I would verify this myself, but I have
never seen that portion of my code actually execute.? So far, it looks
like everything is usually contained within 1 MDL, even though I know it
doesn’t have to be.

Thanks,
* Jim

From: Munafo, Jim (Xetron)
Sent: Thursday, August 13, 2009 2:34 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ethernet Frame Type

Thanks Tom,

Right now I get passed a NET_BUFFER_LIST.? There is only 1 NET_BUFFER in
this list, and there is only 1 MDL in that NET_BUFFER.? MappedSystemVa in
that MDL points to the beginning of the ARP header.

I think I just solved my problem though, and I’m probably just being an
idiot today.? I’m looking at the following in a TCP/IP book:

ARP Packet Format:
|Ethernet Header|??? ???|ARP
Request/Reply|
[eth dest addr][eth src addr][type]??? [hard type][prot type]
[…]

I just looked at the section name (ARP Packet Format) and assumed that what
I was looking at was the ARP packet format.? However, it clearly indicates
that I’m looking at the Ethernet Header in the image of the format in the
book right before the ARP Request/Reply.

Don’t even know what to say.? Thanks for the help though.
* Jim

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Thomas F. Divine
Sent: Thursday, August 13, 2009 2:07 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ethernet Frame Type

Jim,

You do understand that each NET_BUFFER carries a linked-list of MDLs - not
just a single MDL. You can’t make assumptions about the organization of the
singly-linked list of MDLs other than you should expect the unexpected.

To access data in a NET_BUFFER you must write code that walks the MDL chain
to the desired offset. For example, the first MDL may be only 14 bytes long
(ignoring backfill, etc…) and contain the Ethernet header. The next MDL
may
contain the IP header, the next the TCP header, and finally a MDL ( or
more…) containing TCP data.

It’s a little tedious but just write code that “walks the chain” until you
finally reach a MDL that contains data at the offset of interest.

You can assume that headers are not split in the middle of a MDL.

Good luck,

Thomas F. Divine

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Munafo, Jim (Xetron)
Sent: Thursday, August 13, 2009 1:39 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Ethernet Frame Type

I’m currently looking at a NET_BUFFER, and trying to determine the type of
frame that exist in the MDL.? For example, I can get an ARP packet by doing
the following:
ULONG offset;
PMDL currentMDL;
PUCHAR packet;
offset = someNetBufferThatWasPassedToAFunction->DataOffset;
currentMDL = someNetBufferThatWasPassedToAFunction->CurrentMdl;
packet = (PUCHAR) currentMDL->MappedSystemVa;
packet += offset;
From here, packet points to the correct ARP packet, and I can look at
anything I need in the ARP packet.? The problem I am having is how do I know
that this was an ARP packet and not something else (IP Packet).? I suppose I
can look 12 bytes into the packet and see if bytes 13 & 14 are 0x0806
(indicating an ARP packet in the ARP packet format), but this is ghetto.
What I really want is the location of the Ethernet encapsulated header.?
That way, I could just look at the type field in the Ethernet frame and
determine the type of packet.? However, I can’t find this in the MDL.? I
looked at all of the bytes from the StartVa to the MappedSystemVa and none
of them look like the Ethernet frame.
Does anybody know where that is located or have a way of determining the
type of frame from a NET_BUFFER?
Thanks,
T??? Jim

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

OSR_Community_User · October 29, 2009, 11:29am

Thanks for the clarification again Dave. Appreciate it.
– Jim

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of David R. Cattley
Sent: Thursday, October 29, 2009 11:16 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ethernet Frame Type