Force checksum offloading

Hi.

I’m developing an NDIS protocol driver where I send modified Ethernet frames captured at the firewall. My problem is that if the network adapter is configured to offload computation of checksums to the hardware, some frames are intercepted at the firewall with the checksums set to zero, and when I send these frames the NIC doesn’t fill in the correct checksums and the network hardware ends up dropping the traffic.

Is there any way to force the NIC to compute the checksums, even when sending raw Ethernet frames?

Thanks.

You need to set the proper flags in the NBL/NB structures.

Thanks for your reply.
Do you mean the flags in NET_BUFFER_LIST_INFO(nbl, TcpIpChecksumNetBufferListInfo)? I’ve done that already but it hasn’t made any difference:

NDIS_TCP_IP_CHECKSUM_NET_BUFFER_LIST_INFO offload_info;
offload_info.Value = NET_BUFFER_LIST_INFO(nbl, TcpIpChecksumNetBufferListInfo);
offload_info.Transmit.IpHeaderChecksum = 1;
offload_info.Transmit.TcpChecksum = 1;
offload_info.Transmit.UdpChecksum = 1;
NET_BUFFER_LIST_INFO(nbl, TcpIpChecksumNetBufferListInfo) = offload_info.Value;

You can’t “force” the NIC to do checksum calculation, the NIC can offer to do it with certain constraints, and the TCP stack will ask the NIC to take care of if those constraints are met. Your filter needs to assure it doesn’t change packets such that they violate the agreed-on constraints. For example, some NICs can do TCP checksum, if there are no option headers, but if there are option headers, the transport will have to do the checksum in software. If your filter stuffs IP or TCP option headers into the packet, you may then be responsible for the FULL checksum calculation.

Does your filter change the header? NIC TCP checksum offload assumes the TCP/UDP pseudo header (which has the IP addresses and the IP datagram size, see the TCP/UDP offload detailed docs) has been precalculated by the transport, and the result stored in the checksum field. This is so the NIC doesn’t have to parse headers before the TCP/UDP header to calculate the checksum. You also would have to do pseudo header precalculation for both IPv4 and IPv6.

Also note you will see huge (64K or more) LSO TCP packets, which will be segmented by the NIC and TCP checksums calculated after segmentation. Fortunately, the pseudo header is the same for each segment, so you can still precalcualte the checksum.

Some NICs also can do checksum calculation on tunnel encapsulated packets (NVGRE), so if you change the packet you may have to recalculate the pseudo headers for both the tunnel header and the body header, read the NVGRE checksum offload docs. If you insert any new headers, you also have to adjust the TCP header offset in the metadata, as the NIC doesn’t parse the whole packet on Tx, it just calculates the Tx checksum based on knowing the offset of the TCP header. You will need a NIC that does tunnel checksum offload to test this.

Jan

On 11/24/16, 12:48 PM, “xxxxx@lists.osr.com on behalf of xxxxx@nektra.com” wrote:

Thanks for your reply.
Do you mean the flags in NET_BUFFER_LIST_INFO(nbl, TcpIpChecksumNetBufferListInfo)? I’ve done that already but it hasn’t made any difference:

NDIS_TCP_IP_CHECKSUM_NET_BUFFER_LIST_INFO offload_info;
offload_info.Value = NET_BUFFER_LIST_INFO(nbl, TcpIpChecksumNetBufferListInfo);
offload_info.Transmit.IpHeaderChecksum = 1;
offload_info.Transmit.TcpChecksum = 1;
offload_info.Transmit.UdpChecksum = 1;
NET_BUFFER_LIST_INFO(nbl, TcpIpChecksumNetBufferListInfo) = offload_info.Value;


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:>

My driver only modifies the source and destination MAC in the Ethernet
header, and the rest of the frame is left unaltered.
From what I can see, at the time I capture them the IP checksums are
completely unset and the TCP pre-calculations are set (if the frame
contains a TCP packet). So, assuming the OS was about to send an
otherwise valid packet, what would I need to do in my code before
calling NdisSendNetBufferLists()? Just do the IP pre-calculations and
set the flags I mentioned earlier? Are the flags I used sufficient or
did I miss anything?

You can’t “force” the NIC to do checksum calculation, the NIC can offer to do it with certain constraints, and the TCP stack will ask the NIC to take care of if those constraints are met. Your filter needs to assure it doesn’t change packets such that they violate the agreed-on constraints. For example, some NICs can do TCP checksum, if there are no option headers, but if there are option headers, the transport will have to do the checksum in software. If your filter stuffs IP or TCP option headers into the packet, you may then be responsible for the FULL checksum calculation.

Does your filter change the header? NIC TCP checksum offload assumes the TCP/UDP pseudo header (which has the IP addresses and the IP datagram size, see the TCP/UDP offload detailed docs) has been precalculated by the transport, and the result stored in the checksum field. This is so the NIC doesn’t have to parse headers before the TCP/UDP header to calculate the checksum. You also would have to do pseudo header precalculation for both IPv4 and IPv6.

Also note you will see huge (64K or more) LSO TCP packets, which will be segmented by the NIC and TCP checksums calculated after segmentation. Fortunately, the pseudo header is the same for each segment, so you can still precalcualte the checksum.

Some NICs also can do checksum calculation on tunnel encapsulated packets (NVGRE), so if you change the packet you may have to recalculate the pseudo headers for both the tunnel header and the body header, read the NVGRE checksum offload docs. If you insert any new headers, you also have to adjust the TCP header offset in the metadata, as the NIC doesn’t parse the whole packet on Tx, it just calculates the Tx checksum based on knowing the offset of the TCP header. You will need a NIC that does tunnel checksum offload to test this.

Jan

On 11/24/16, 12:48 PM, “xxxxx@lists.osr.com on behalf of xxxxx@nektra.com” wrote:
>
> Thanks for your reply.
> Do you mean the flags in NET_BUFFER_LIST_INFO(nbl, TcpIpChecksumNetBufferListInfo)? I’ve done that already but it hasn’t made any difference:
>
> NDIS_TCP_IP_CHECKSUM_NET_BUFFER_LIST_INFO offload_info;
> offload_info.Value = NET_BUFFER_LIST_INFO(nbl, TcpIpChecksumNetBufferListInfo);
> offload_info.Transmit.IpHeaderChecksum = 1;
> offload_info.Transmit.TcpChecksum = 1;
> offload_info.Transmit.UdpChecksum = 1;
> NET_BUFFER_LIST_INFO(nbl, TcpIpChecksumNetBufferListInfo) = offload_info.Value;
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at http:
>
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></http:></http:></http:>

There are a couple of flags, that specify IPv4 or Ipv6, and a flag for TCP. There also is an offset field telling the NIC where in the packet the TCP header is. These should all be set in the original NBL and assuming you are not passing down the original NBL calling NdisCopySendNetBufferListInfo should copy them. If you only are changing the MAC addresses, you should not need to recalculate the pseudo header.

A little gotcha about modifying packets is Windows will share header memory fragments between packets that have the same value, or at least the driver certifications tests do, so if you modify a packet you must allocate new memory for the changed data. If you don’t allocate new memory, when you modify it, you also unexpected modify the header for some other packet. These other packets may already have been passed to the NIC but are not yet send by the hardware, so the packet may have looked correct on a send side capture, but what the hardware sees is not what you expected. I discovered this fun fact while running the certification tests for checksum offload on a NIC that calculated the checksum in the driver sometimes. Even though the NBL flag NDIS_NBL_FLAGS_SEND_READ_ONLY is not set, the packets are read only.

A useful debugging strategy is also to manually disable the NIC checksum calculations, like turn off the Tx IP checksum offload and see if the problem goes away, or turn off the TCP offload and check it. Capturing on the Rx side, or with a switch set to monitor the Tx side of a port can also give the most accurate sniffing. Also note that wireshark and the Microsoft sniffers capture packets differently. I tend to use the Microsoft sniffer (Network Monitor 3.4), which captures better, although Message Analyzer 1.4 analyzes protocols better.

Jan

On 11/25/16, 5:25 AM, “xxxxx@lists.osr.com on behalf of Víctor M. González” wrote:

My driver only modifies the source and destination MAC in the Ethernet
header, and the rest of the frame is left unaltered.
From what I can see, at the time I capture them the IP checksums are
completely unset and the TCP pre-calculations are set (if the frame
contains a TCP packet). So, assuming the OS was about to send an
otherwise valid packet, what would I need to do in my code before
calling NdisSendNetBufferLists()? Just do the IP pre-calculations and
set the flags I mentioned earlier? Are the flags I used sufficient or
did I miss anything?

> You can’t “force” the NIC to do checksum calculation, the NIC can offer to do it with certain constraints, and the TCP stack will ask the NIC to take care of if those constraints are met. Your filter needs to assure it doesn’t change packets such that they violate the agreed-on constraints. For example, some NICs can do TCP checksum, if there are no option headers, but if there are option headers, the transport will have to do the checksum in software. If your filter stuffs IP or TCP option headers into the packet, you may then be responsible for the FULL checksum calculation.
>
> Does your filter change the header? NIC TCP checksum offload assumes the TCP/UDP pseudo header (which has the IP addresses and the IP datagram size, see the TCP/UDP offload detailed docs) has been precalculated by the transport, and the result stored in the checksum field. This is so the NIC doesn’t have to parse headers before the TCP/UDP header to calculate the checksum. You also would have to do pseudo header precalculation for both IPv4 and IPv6.
>
> Also note you will see huge (64K or more) LSO TCP packets, which will be segmented by the NIC and TCP checksums calculated after segmentation. Fortunately, the pseudo header is the same for each segment, so you can still precalcualte the checksum.
>
> Some NICs also can do checksum calculation on tunnel encapsulated packets (NVGRE), so if you change the packet you may have to recalculate the pseudo headers for both the tunnel header and the body header, read the NVGRE checksum offload docs. If you insert any new headers, you also have to adjust the TCP header offset in the metadata, as the NIC doesn’t parse the whole packet on Tx, it just calculates the Tx checksum based on knowing the offset of the TCP header. You will need a NIC that does tunnel checksum offload to test this.
>
> Jan
>
> On 11/24/16, 12:48 PM, “xxxxx@lists.osr.com on behalf of xxxxx@nektra.com” wrote:
>
> Thanks for your reply.
> Do you mean the flags in NET_BUFFER_LIST_INFO(nbl, TcpIpChecksumNetBufferListInfo)? I’ve done that already but it hasn’t made any difference:
>
> NDIS_TCP_IP_CHECKSUM_NET_BUFFER_LIST_INFO offload_info;
> offload_info.Value = NET_BUFFER_LIST_INFO(nbl, TcpIpChecksumNetBufferListInfo);
> offload_info.Transmit.IpHeaderChecksum = 1;
> offload_info.Transmit.TcpChecksum = 1;
> offload_info.Transmit.UdpChecksum = 1;
> NET_BUFFER_LIST_INFO(nbl, TcpIpChecksumNetBufferListInfo) = offload_info.Value;
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at http:
>
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at http:


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></http:></http:></http:></http:></http:></http:>

Yeah, I allocate a temporary buffer to hold the frame copy before I send
it, and this copy is what I modify.

It turns out that those flags from earlier were correct (more or less).
I initially didn’t notice any improvements because with my old code I
was getting inbound frames with the correct checksum (because they had
been generated by the remote NIC) and outbound frames with zero. I
failed to notice that when I turned the flags on this had reversed. When
I did realize it, I understood that it’s because the checksum is the
additive inverse of a one’s complement sum of 16-bit integers.
So basically all I need to do is turn the flags on and make sure the IP
checksum field is reset for outbound frames, and for inbound frames just
leave them off and send the frame as it is.

Thanks for the help, guys.

There are a couple of flags, that specify IPv4 or Ipv6, and a flag for TCP. There also is an offset field telling the NIC where in the packet the TCP header is. These should all be set in the original NBL and assuming you are not passing down the original NBL calling NdisCopySendNetBufferListInfo should copy them. If you only are changing the MAC addresses, you should not need to recalculate the pseudo header.

A little gotcha about modifying packets is Windows will share header memory fragments between packets that have the same value, or at least the driver certifications tests do, so if you modify a packet you must allocate new memory for the changed data. If you don’t allocate new memory, when you modify it, you also unexpected modify the header for some other packet. These other packets may already have been passed to the NIC but are not yet send by the hardware, so the packet may have looked correct on a send side capture, but what the hardware sees is not what you expected. I discovered this fun fact while running the certification tests for checksum offload on a NIC that calculated the checksum in the driver sometimes. Even though the NBL flag NDIS_NBL_FLAGS_SEND_READ_ONLY is not set, the packets are read only.

A useful debugging strategy is also to manually disable the NIC checksum calculations, like turn off the Tx IP checksum offload and see if the problem goes away, or turn off the TCP offload and check it. Capturing on the Rx side, or with a switch set to monitor the Tx side of a port can also give the most accurate sniffing. Also note that wireshark and the Microsoft sniffers capture packets differently. I tend to use the Microsoft sniffer (Network Monitor 3.4), which captures better, although Message Analyzer 1.4 analyzes protocols better.

Jan

On 11/25/16, 5:25 AM, “xxxxx@lists.osr.com on behalf of Víctor M. González” wrote:
>
> My driver only modifies the source and destination MAC in the Ethernet
> header, and the rest of the frame is left unaltered.
> From what I can see, at the time I capture them the IP checksums are
> completely unset and the TCP pre-calculations are set (if the frame
> contains a TCP packet). So, assuming the OS was about to send an
> otherwise valid packet, what would I need to do in my code before
> calling NdisSendNetBufferLists()? Just do the IP pre-calculations and
> set the flags I mentioned earlier? Are the flags I used sufficient or
> did I miss anything?
>
>
> > You can’t “force” the NIC to do checksum calculation, the NIC can offer to do it with certain constraints, and the TCP stack will ask the NIC to take care of if those constraints are met. Your filter needs to assure it doesn’t change packets such that they violate the agreed-on constraints. For example, some NICs can do TCP checksum, if there are no option headers, but if there are option headers, the transport will have to do the checksum in software. If your filter stuffs IP or TCP option headers into the packet, you may then be responsible for the FULL checksum calculation.
> >
> > Does your filter change the header? NIC TCP checksum offload assumes the TCP/UDP pseudo header (which has the IP addresses and the IP datagram size, see the TCP/UDP offload detailed docs) has been precalculated by the transport, and the result stored in the checksum field. This is so the NIC doesn’t have to parse headers before the TCP/UDP header to calculate the checksum. You also would have to do pseudo header precalculation for both IPv4 and IPv6.
> >
> > Also note you will see huge (64K or more) LSO TCP packets, which will be segmented by the NIC and TCP checksums calculated after segmentation. Fortunately, the pseudo header is the same for each segment, so you can still precalcualte the checksum.
> >
> > Some NICs also can do checksum calculation on tunnel encapsulated packets (NVGRE), so if you change the packet you may have to recalculate the pseudo headers for both the tunnel header and the body header, read the NVGRE checksum offload docs. If you insert any new headers, you also have to adjust the TCP header offset in the metadata, as the NIC doesn’t parse the whole packet on Tx, it just calculates the Tx checksum based on knowing the offset of the TCP header. You will need a NIC that does tunnel checksum offload to test this.
> >
> > Jan
> >
> > On 11/24/16, 12:48 PM, “xxxxx@lists.osr.com on behalf of xxxxx@nektra.com” wrote:
> >
> > Thanks for your reply.
> > Do you mean the flags in NET_BUFFER_LIST_INFO(nbl, TcpIpChecksumNetBufferListInfo)? I’ve done that already but it hasn’t made any difference:
> >
> > NDIS_TCP_IP_CHECKSUM_NET_BUFFER_LIST_INFO offload_info;
> > offload_info.Value = NET_BUFFER_LIST_INFO(nbl, TcpIpChecksumNetBufferListInfo);
> > offload_info.Transmit.IpHeaderChecksum = 1;
> > offload_info.Transmit.TcpChecksum = 1;
> > offload_info.Transmit.UdpChecksum = 1;
> > NET_BUFFER_LIST_INFO(nbl, TcpIpChecksumNetBufferListInfo) = offload_info.Value;
> >
> > —
> > NTDEV is sponsored by OSR
> >
> > Visit the list online at: http:
> >
> > MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> > Details at http:
> >
> > To unsubscribe, visit the List Server section of OSR Online at http:
> >
> >
> >
> > —
> > NTDEV is sponsored by OSR
> >
> > Visit the list online at: http:
> >
> > MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> > Details at http:
> >
> > To unsubscribe, visit the List Server section of OSR Online at http:
>
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at http:
>
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:></http:>