WFP callout gets out-of-order TCP packets

Save_me_O_God

Any suggestions or ideas will be appreciated. I am willing to test out anything and provide extra information when needed.

I'm writing a VPN driver that (in the outbound direction)
 1. takes up the packets in FWPM_LAYER_OUTBOUND_IPPACKET_V4
 2. encrypts each packet (including the IP header)
 3. adds a UDP header and a new IP header that points to the VPN peer

The inbound direction does the reverse
 1. takes up the packets in FWPM_LAYER_INBOUND_IPPACKET_V4
 2. decrypts the data
 3. inject the decrypted packet back using FwpsInjectNetworkReceiveAsync0

Both directions work. While the downloading speed is perfect,
the uploading speed (outbound direction) gets extremely slow when the driver handles TCP-based file transfer, for example, SCP.

After some investigation, I realized that in the outbound direction, the packets come in an out-of-order manner.
 1. Whenever the callout gets called, I log the TCP sequence, smaller sequences might come later.
 2. WireShark on the receiving machine shows lots of out-of-order packets.
 *. This (out-of-order packets) slows down the uploading process.

As a last attempt, I've tested WFPSampler, and as it turns out,
its callout miraculously gets the packets in an ordered manner (in the outbound direction).
Thus, my issue doesn't appear in WFPSampler.

And here's the question:

• is there anything I can do to make the packets ordered when they come to my callout?
• or is it abnormal that my callout sees out-of-order packets?

Save_me_O_God

The outbound (upload) direction looks something like this.
This is captured on the receiver machine (172.17.0.3)

Save_me_O_God

Solved, see: https://community.osr.com/discussion/293931/wfpsampler-callout-doesnt-get-all-the-packets