Virtual NIC and virtual bus using KMDF+NDIS vs WDM+NDIS

NTDEV
1

The team I’m on is doing an NDIS 6.2 virtual NIC driver on a virtual bus. On
checking the winqual site, it says a NIC driver can ONLY make calls to NDIS
or WDF. Another team member has expressed the view that in the past he wrote
virtual NIC’s that called WDM API’s, and there is no advantage to using WDF.
This product also has a virtual bus, which I personally think we should use
KDMF for (it would likely be a bus filter, not unlike the KMDF kbfiltr
sample in the WDK), and the other developer’s view is the old tried and true
way of doing a bus driver in WDM is fine. In the past I too have written
virtual NIC and busses using the WDM API, and my experience was it was a lot
of tough debugging and testing to actually get this to work correctly. All
the PnP and power management states are no exactly perfectly documented, and
you need to have a QA group that can exercise the PnP/Power nooks and
crannies.

As this list has an excellent group of highly experienced developer,
including folks from Microsoft like Doron and Jake, I’m interested in
people’s opinions. I do know a virtual NIC and bus driver can be done
without KDMF, but with KMDF being a pretty mature product now, it seems like
the optimal choice from a stability and development efficiency viewpoint
would be to use KMDF where possible. I’d also hate to get the product
written, and find we can’t pass WHQL certification because our virtual NIC
makes other than NDIS or WDF calls, a risk easily avoided.

Basically my manger (not a developer) needs input to decide which of her two
highly experienced Windows developers she should side with.

Your feedback is appreciated!

Jan

2

When you say bus filter, what exactly do you mean functionally? Classic “viral” bus filters that filter at the root and dynamically attach to PDOs enumerated via QDR, thus adding themselves deeper in the stack at runtime is not supported by KMDF. OTOH, if you write an NDIS-WDM driver, KMDF adds value if you are talking to a usb nic (not this instance) or your IO interface to the PDO is irp based and you have to track sent IO, in which case WDFIOTARGETs are very very useful

d

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Jan Bottorff
Sent: Tuesday, December 06, 2011 11:15 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Virtual NIC and virtual bus using KMDF+NDIS vs WDM+NDIS

The team I’m on is doing an NDIS 6.2 virtual NIC driver on a virtual bus. On checking the winqual site, it says a NIC driver can ONLY make calls to NDIS or WDF. Another team member has expressed the view that in the past he wrote virtual NIC’s that called WDM API’s, and there is no advantage to using WDF. This product also has a virtual bus, which I personally think we should use KDMF for (it would likely be a bus filter, not unlike the KMDF kbfiltr sample in the WDK), and the other developer’s view is the old tried and true way of doing a bus driver in WDM is fine. In the past I too have written virtual NIC and busses using the WDM API, and my experience was it was a lot of tough debugging and testing to actually get this to work correctly. All the PnP and power management states are no exactly perfectly documented, and you need to have a QA group that can exercise the PnP/Power nooks and crannies.

As this list has an excellent group of highly experienced developer, including folks from Microsoft like Doron and Jake, I’m interested in people’s opinions. I do know a virtual NIC and bus driver can be done without KDMF, but with KMDF being a pretty mature product now, it seems like the optimal choice from a stability and development efficiency viewpoint would be to use KMDF where possible. I’d also hate to get the product written, and find we can’t pass WHQL certification because our virtual NIC makes other than NDIS or WDF calls, a risk easily avoided.

Basically my manger (not a developer) needs input to decide which of her two highly experienced Windows developers she should side with.

Your feedback is appreciated!

Jan

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

3

Hi Doron,

When I say bus filter, I mean a filter on a lower FDO that is NOT a bus.
Actually, the lower device is a standard Ethernet NIC, and there would be an
NDIS protocol interface used to get packets to/from the virtual NIC’s tp the
lower physical NIC. The reason we want this architecture over an IM driver
is so we can get things like iSCSI booting or software FCoE to work for the
child devices. An IM driver seemed problematic for boot and potentially
things like RDMA which I assume will need the DMA adapter address mapping
interface to work. In an IM driver the virtual NICs are all root enumerated,
so their PnP hierarchy is bogus.

I understand from our past conversations that KMDF can’t be used for a
filter on a lower bus, filtering the PDO’s. Are there official names for the
two flavors of bus filter?

Thanks.

Jan

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Doron Holan
Sent: Tuesday, December 06, 2011 11:20 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Virtual NIC and virtual bus using KMDF+NDIS vs WDM+NDIS

When you say bus filter, what exactly do you mean functionally? Classic
“viral” bus filters that filter at the root and dynamically attach to PDOs
enumerated via QDR, thus adding themselves deeper in the stack at runtime is
not supported by KMDF. OTOH, if you write an NDIS-WDM driver, KMDF adds
value if you are talking to a usb nic (not this instance) or your IO
interface to the PDO is irp based and you have to track sent IO, in which
case WDFIOTARGETs are very very useful

d

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jan Bottorff
Sent: Tuesday, December 06, 2011 11:15 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Virtual NIC and virtual bus using KMDF+NDIS vs WDM+NDIS

The team I’m on is doing an NDIS 6.2 virtual NIC driver on a virtual bus. On
checking the winqual site, it says a NIC driver can ONLY make calls to NDIS
or WDF. Another team member has expressed the view that in the past he wrote
virtual NIC’s that called WDM API’s, and there is no advantage to using WDF.
This product also has a virtual bus, which I personally think we should use
KDMF for (it would likely be a bus filter, not unlike the KMDF kbfiltr
sample in the WDK), and the other developer’s view is the old tried and true
way of doing a bus driver in WDM is fine. In the past I too have written
virtual NIC and busses using the WDM API, and my experience was it was a lot
of tough debugging and testing to actually get this to work correctly. All
the PnP and power management states are no exactly perfectly documented, and
you need to have a QA group that can exercise the PnP/Power nooks and
crannies.

As this list has an excellent group of highly experienced developer,
including folks from Microsoft like Doron and Jake, I’m interested in
people’s opinions. I do know a virtual NIC and bus driver can be done
without KDMF, but with KMDF being a pretty mature product now, it seems like
the optimal choice from a stability and development efficiency viewpoint
would be to use KMDF where possible. I’d also hate to get the product
written, and find we can’t pass WHQL certification because our virtual NIC
makes other than NDIS or WDF calls, a risk easily avoided.

Basically my manger (not a developer) needs input to decide which of her two
highly experienced Windows developers she should side with.

Your feedback is appreciated!

Jan

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

4

You just described a device upper filter. Very easy to do everything you want in kmdf. Your bus driver could also be the upper filter if I understand correctly

d

debt from my phone

From: Jan Bottorff
Sent: 12/6/2011 11:46 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Virtual NIC and virtual bus using KMDF+NDIS vs WDM+NDIS

Hi Doron,

When I say bus filter, I mean a filter on a lower FDO that is NOT a bus. Actually, the lower device is a standard Ethernet NIC, and there would be an NDIS protocol interface used to get packets to/from the virtual NIC?s tp the lower physical NIC. The reason we want this architecture over an IM driver is so we can get things like iSCSI booting or software FCoE to work for the child devices. An IM driver seemed problematic for boot and potentially things like RDMA which I assume will need the DMA adapter address mapping interface to work. In an IM driver the virtual NICs are all root enumerated, so their PnP hierarchy is bogus.

I understand from our past conversations that KMDF can?t be used for a filter on a lower bus, filtering the PDO?s. Are there official names for the two flavors of bus filter?

Thanks.

Jan

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Doron Holan
Sent: Tuesday, December 06, 2011 11:20 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Virtual NIC and virtual bus using KMDF+NDIS vs WDM+NDIS

When you say bus filter, what exactly do you mean functionally? Classic ?viral? bus filters that filter at the root and dynamically attach to PDOs enumerated via QDR, thus adding themselves deeper in the stack at runtime is not supported by KMDF. OTOH, if you write an NDIS-WDM driver, KMDF adds value if you are talking to a usb nic (not this instance) or your IO interface to the PDO is irp based and you have to track sent IO, in which case WDFIOTARGETs are very very useful

d

From: xxxxx@lists.osr.commailto:xxxxx [mailto:xxxxx@lists.osr.com]mailto: On Behalf Of Jan Bottorff
Sent: Tuesday, December 06, 2011 11:15 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Virtual NIC and virtual bus using KMDF+NDIS vs WDM+NDIS

The team I?m on is doing an NDIS 6.2 virtual NIC driver on a virtual bus. On checking the winqual site, it says a NIC driver can ONLY make calls to NDIS or WDF. Another team member has expressed the view that in the past he wrote virtual NIC?s that called WDM API?s, and there is no advantage to using WDF. This product also has a virtual bus, which I personally think we should use KDMF for (it would likely be a bus filter, not unlike the KMDF kbfiltr sample in the WDK), and the other developer?s view is the old tried and true way of doing a bus driver in WDM is fine. In the past I too have written virtual NIC and busses using the WDM API, and my experience was it was a lot of tough debugging and testing to actually get this to work correctly. All the PnP and power management states are no exactly perfectly documented, and you need to have a QA group that can exercise the PnP/Power nooks and crannies.

As this list has an excellent group of highly experienced developer, including folks from Microsoft like Doron and Jake, I?m interested in people?s opinions. I do know a virtual NIC and bus driver can be done without KDMF, but with KMDF being a pretty mature product now, it seems like the optimal choice from a stability and development efficiency viewpoint would be to use KMDF where possible. I?d also hate to get the product written, and find we can?t pass WHQL certification because our virtual NIC makes other than NDIS or WDF calls, a risk easily avoided.

Basically my manger (not a developer) needs input to decide which of her two highly experienced Windows developers she should side with.

Your feedback is appreciated!

Jan

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</mailto:></mailto:xxxxx>

5

Jan,

On 07/12/2011 07:15, Jan Bottorff wrote:

The team I’m on is doing an NDIS 6.2 virtual NIC driver on a virtual
bus. On checking the winqual site, it says a NIC driver can ONLY make
calls to NDIS or WDF. Another team member has expressed the view that
in the past he wrote virtual NIC’s that called WDM API’s, and there is
no advantage to using WDF. This product also has a virtual bus, which
I personally think we should use KDMF for (it would likely be a bus
filter, not unlike the KMDF kbfiltr sample in the WDK), and the other
developer’s view is the old tried and true way of doing a bus driver
in WDM is fine. In the past I too have written virtual NIC and busses
using the WDM API, and my experience was it was a lot of tough
debugging and testing to actually get this to work correctly. All the
PnP and power management states are no exactly perfectly documented,
and you need to have a QA group that can exercise the PnP/Power nooks
and crannies.

We followed the virtual bus driver model for our converged network
adapter. In our case the VBD attaches to the PCI device, the VBD
creates one or more child PDOs (for virtual NICs or other devices) and
our NDIS driver attaches to those. I’d agree that WDF is the right way
to go for the VBD, the toaster bus sample is a good starting point to
learn from. Depending on your design you may find that your NDIS driver
needs to use very little WDF. In our case almost all of the non-NDIS
work is done by our VBD, the NDIS drivers simply query for a private
driver defined interface from the VBD using WdfFdoQueryForInterface().

One of the strengths with WDF as a framework is that you can mix in some
WDM as well for areas not supported via WDF. An example that comes to
mind is setting the target CPU affinity and importance of DPCs for RSS
purposes. Simply call WDF’s WdfDpcWdmGetDpc() and then call the WDMs
KeSetTargetProcessorDpcEx() and KeSetImportanceDpc() DDIs.

The hiccups you’ll come across will be mostly because you are using a
VBD model for a network adapter. These hiccups are the same
irrespective of the choice between WDF or WDM. (Some) things that come
to mind:

  1. You can’t implement NDIS Receive Side Throttling
    (http://msdn.microsoft.com/en-us/library/windows/hardware/ff570441(v=VS.85).aspx)
    as NDIS doesn’t handle your devices interrupts or DPCs.
    Although you can implement a scheme to avoid your DPCs taking up too
    much time. See KeQueryDpcWatchdogInformation()
    (http://msdn.microsoft.com/en-us/library/windows/hardware/ff553005(v=vs.85).aspx),
    the WDK’s netvmini sample (src\network\ndis\netvmini\6x) implements
    a DPC watchdog avoidance scheme using this DDI
  2. Calls to read or adjust PCI configuration space (i.e.
    NdisMGetBusData(), NdisMSetBusData(), NdisMConfigMSIXTableEntry(),
    etc.) need to be replaced with calls to the underlying PCI
    interfaces via your VBD (BUS_INTERFACE_STANDARD and
    MSIX_TABLE_CONFIG_INTERFACE).
    See
    http://msdn.microsoft.com/en-us/library/windows/hardware/ff564581(v=VS.85).aspx
  3. The WLK’s CheckINF test fails to understand the VBD model, so you’ll
    need to apply an errata to any WLK submissions:
    https://winqual.microsoft.com/ec/ECDetails.aspx?id=2958
  4. Depending on your design your NDIS driver may not be making
    scatter/gather DMA calls through the NDIS DDIs as your DMA being
    handled by your VBD driver instead. If so another WLK errata you’ll
    need is:
    https://winqual.microsoft.com/ec/ECDetails.aspx?id=2531
  5. Regarding iSCSI boot support:
  6. You’ll want your VBD driver service set to boot start in its INF
    file. Although the the iSCSI boot process will cope with auto
    start NDIS drivers, it doesn’t follow the chain of drivers and
    devices needed to support the NDIS device (i.e. your VBD) and
    pre-load those.
  7. You need to enable special file support on your device and any
    child devices your VBD creates. With WDF you’ll find you need
    very little code to do this, see
    http://msdn.microsoft.com/en-us/library/windows/hardware/ff544696(v=vs.85).aspx.
  8. We also found we needed to implement the PNP_LOCATION_INTERFACE
    (see
    http://msdn.microsoft.com/en-us/library/windows/hardware/ff546564(v=VS.85).aspx)
    within the VBD for its child devices to make iSCSI boot work,
    your mileage may vary.

One pain with WDF is loading your driver during OS installation, for
example when wanting to install the OS to an iSCSI target reachable via
your network adapter. WinPE (the environment used during OS
installation) doesn’t support enough of the Windows Update hot-fix
framework to allow the WDF co-installer to update its “in-box” version
of WDF. So for Vista’s PE you need to use WDF 1.5, WDF 1.7 for Server
2008’s PE and WDF 1.9 for Server 2008 R2’s PE. The PE image used for OS
installation could be customised to include the correct version of WDF
in advance, but that loses the simplicity of clicking the Load Drivers
button (or pressing F6 at the right time) and supplying your drivers via
some other media.

Best regards,

Andrew

–
Andrew Lee Solarflare Communications
mailto:xxxxx@solarflare.com http://www.solarflare.com/

6

I have to say it’s indeed very well summarized and written! Definitely from
someone who has been there and done that.

One of the obvious thing that I want to add: To support iSCSI boot, you
will need OptionROM support and a Windows crashdump driver that doesn’t
rely on your NDIS/VBD combo or storport mini/VBD. The crashdump driver is a
*lot* of hard work. You can’t hibernate to SAN via ndis boot path (msiscsi
initiator). This has nothing to do with what driver model you are using
though.

Calvin Guan
Broadcom Corp. Former C-NIC Windows Driver Progammer

On Wed, Dec 7, 2011 at 7:31 AM, Andrew Lee wrote:

> Jan,
>
>
> On 07/12/2011 07:15, Jan Bottorff wrote:
>
> The team I?m on is doing an NDIS 6.2 virtual NIC driver on a virtual
> bus. On checking the winqual site, it says a NIC driver can ONLY make calls
> to NDIS or WDF. Another team member has expressed the view that in the past
> he wrote virtual NIC?s that called WDM API?s, and there is no advantage to
> using WDF. This product also has a virtual bus, which I personally think we
> should use KDMF for (it would likely be a bus filter, not unlike the KMDF
> kbfiltr sample in the WDK), and the other developer?s view is the old tried
> and true way of doing a bus driver in WDM is fine. In the past I too have
> written virtual NIC and busses using the WDM API, and my experience was it
> was a lot of tough debugging and testing to actually get this to work
> correctly. All the PnP and power management states are no exactly perfectly
> documented, and you need to have a QA group that can exercise the PnP/Power
> nooks and crannies.
>
> We followed the virtual bus driver model for our converged network
> adapter. In our case the VBD attaches to the PCI device, the VBD creates
> one or more child PDOs (for virtual NICs or other devices) and our NDIS
> driver attaches to those. I’d agree that WDF is the right way to go for
> the VBD, the toaster bus sample is a good starting point to learn from.
> Depending on your design you may find that your NDIS driver needs to use
> very little WDF. In our case almost all of the non-NDIS work is done by
> our VBD, the NDIS drivers simply query for a private driver defined
> interface from the VBD using WdfFdoQueryForInterface().
>
> One of the strengths with WDF as a framework is that you can mix in some
> WDM as well for areas not supported via WDF. An example that comes to mind
> is setting the target CPU affinity and importance of DPCs for RSS
> purposes. Simply call WDF’s WdfDpcWdmGetDpc() and then call the WDMs
> KeSetTargetProcessorDpcEx() and KeSetImportanceDpc() DDIs.
>
> The hiccups you’ll come across will be mostly because you are using a VBD
> model for a network adapter. These hiccups are the same irrespective of
> the choice between WDF or WDM. (Some) things that come to mind:
>
> 1. You can’t implement NDIS Receive Side Throttling (
> http://msdn.microsoft.com/en-us/library/windows/hardware/ff570441(v=VS.85).aspx)
> as NDIS doesn’t handle your devices interrupts or DPCs.
> Although you can implement a scheme to avoid your DPCs taking up too
> much time. See KeQueryDpcWatchdogInformation() (
> http://msdn.microsoft.com/en-us/library/windows/hardware/ff553005(v=vs.85).aspx),
> the WDK’s netvmini sample (src\network\ndis\netvmini\6x) implements a DPC
> watchdog avoidance scheme using this DDI
> 2. Calls to read or adjust PCI configuration space (i.e.
> NdisMGetBusData(), NdisMSetBusData(), NdisMConfigMSIXTableEntry(), etc.)
> need to be replaced with calls to the underlying PCI interfaces via your
> VBD (BUS_INTERFACE_STANDARD and MSIX_TABLE_CONFIG_INTERFACE).
> See
> http://msdn.microsoft.com/en-us/library/windows/hardware/ff564581(v=VS.85).aspx
> 3. The WLK’s CheckINF test fails to understand the VBD model, so
> you’ll need to apply an errata to any WLK submissions:
> https://winqual.microsoft.com/ec/ECDetails.aspx?id=2958
> 4. Depending on your design your NDIS driver may not be making
> scatter/gather DMA calls through the NDIS DDIs as your DMA being handled by
> your VBD driver instead. If so another WLK errata you’ll need is:
> https://winqual.microsoft.com/ec/ECDetails.aspx?id=2531
> 5. Regarding iSCSI boot support:
> 1. You’ll want your VBD driver service set to boot start in its INF
> file. Although the the iSCSI boot process will cope with auto start NDIS
> drivers, it doesn’t follow the chain of drivers and devices needed to
> support the NDIS device (i.e. your VBD) and pre-load those.
> 2. You need to enable special file support on your device and any
> child devices your VBD creates. With WDF you’ll find you need very little
> code to do this, see
> http://msdn.microsoft.com/en-us/library/windows/hardware/ff544696(v=vs.85).aspx
> .
> 3. We also found we needed to implement the PNP_LOCATION_INTERFACE
> (see
> http://msdn.microsoft.com/en-us/library/windows/hardware/ff546564(v=VS.85).aspx)
> within the VBD for its child devices to make iSCSI boot work, your mileage
> may vary.
>
> One pain with WDF is loading your driver during OS installation, for
> example when wanting to install the OS to an iSCSI target reachable via
> your network adapter. WinPE (the environment used during OS installation)
> doesn’t support enough of the Windows Update hot-fix framework to allow the
> WDF co-installer to update its “in-box” version of WDF. So for Vista’s PE
> you need to use WDF 1.5, WDF 1.7 for Server 2008’s PE and WDF 1.9 for
> Server 2008 R2’s PE. The PE image used for OS installation could be
> customised to include the correct version of WDF in advance, but that loses
> the simplicity of clicking the Load Drivers button (or pressing F6 at the
> right time) and supplying your drivers via some other media.
>
> Best regards,
>
>
> Andrew
>
> –
> Andrew Lee Solarflare Communicationsmailto:xxxxx@solarflare.com http://www.solarflare.com/
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

7

On 07/12/2011 19:27, Calvin Guan (news) wrote:

I have to say it’s indeed very well summarized and written! Definitely
from someone who has been there and done that.
One of the obvious thing that I want to add: To support iSCSI boot,
you will need OptionROM support and a Windows crashdump driver that
doesn’t rely on your NDIS/VBD combo or storport mini/VBD. The
crashdump driver is a *lot* of hard work. You can’t hibernate to SAN
via ndis boot path (msiscsi initiator).
Yes, I completely agree. Writing the iSCSI boot option ROM support and
an iSCSI crash dump driver is not a small undertaking. Depending on the
capabilities of your network adapter hardware they both may need to
include an implementation of an iSCSI initiator, TCP/IP stack and
network adapter driver. In the case of crashdump, all written to cope
with the rather limited STORPORT environment available to drivers after
a crash.

As far as I am aware iSCSI crashdump support is still not documented in
the WDK and is only available from Microsoft under an NDA.

Best regards,

Andrew

–
Andrew Lee Solarflare Communications
mailto:xxxxx@solarflare.com http://www.solarflare.com/