ndis signalling that underlying hardware has changed

Under Xen it is possible to ‘live migrate’ a virtual machine from one
physical machine to another. On startup, I get info about the
acceleration features that my virtual network card supports - scatter
gather, checksum offload, and large send offload. When doing a live
migration, it is possible that the new physical machine supports a
different feature set than the old physical machine. If this is the
case, how should I tell NDIS? The last resort would be to simulate an
unplug and replug of the device so that Windows will start it from the
start, but maybe there is a better way? I wouldn’t be surprised if there
isn’t though as this isn’t a situation that the designers would have had
in mind when NDIS was put together…

The device unplug and replug is not really desirable as apart from a
slight performance hit while the migration is happening and then a pause
of a few milliseconds while the transfer is committed, the user isn’t
supposed to be able to tell that this has happened. A difference in
network acceleration features is going to be uncommon though.

Thanks

James

I think your choices are to either only support a common set of
features or simulate unplug/plug on mismatch.

On Saturday, August 14, 2010, James Harper
wrote:
> Under Xen it is possible to ‘live migrate’ a virtual machine from one
> physical machine to another. On startup, I get info about the
> acceleration features that my virtual network card supports - scatter
> gather, checksum offload, and large send offload. When doing a live
> migration, it is possible that the new physical machine supports a
> different feature set than the old physical machine. If this is the
> case, how should I tell NDIS? The last resort would be to simulate an
> unplug and replug of the device so that Windows will start it from the
> start, but maybe there is a better way? I wouldn’t be surprised if there
> isn’t though as this isn’t a situation that the designers would have had
> in mind when NDIS was put together…
>
> The device unplug and replug is not really desirable as apart from a
> slight performance hit while the migration is happening and then a pause
> of a few milliseconds while the transfer is committed, the user isn’t
> supposed to be able to tell that this has happened. A difference in
> network acceleration features is going to be uncommon though.
>
> Thanks
>
> James
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>


Mark Roddy

This can be done with an NDIS6 miniport.

When initializing your adapter, claim hardware capabilities for every offload that your driver can perform, even if the *current* underlying physical host can’t do them. [NdisMSetMiniportAttributes(NDIS_MINIPORT_ADAPTER_OFFLOAD_ATTRIBUTES.HardwareOffloadCapabilities)]. Reflect the *current* host capabilities in DefaultOffloadConfiguration.

Then, as your underlying host changes (and offload capabilities change), indicate up a NDIS_STATUS_TASK_OFFLOAD_CURRENT_CONFIG via NdisMIndicateStatusEx with the new checksum offload and large send offload capabilities. TCPIP will react to it accordingly.

If you think about it for a moment, you will probably notice that there is necessarily a race window where you have just finished migrating to a new host that does not support some offload, and your status indication hasn’t gone up yet. This is ok – just indicate receive packets without any offload performed, and TCPIP will fall back to computing checksums on the CPU. You can temporarily fail send packets if they have an incompatible offload. (Protocols do have to cope with a few dropped packets now and then, and VM migration is hopefully an infrequent trauma). If the migration is slow-ish, you might want to consider popping a link state indication with mediadisconnect for the duration, so that overlying protocols understand that things are spotty.

I’m not quite sure what you mean w.r.t. scatter-gather changing – I don’t see how details of the SG list mapping really affects a guest in a VM. At that point, maybe you should just eject the device and re-enumerate it, since that is likely something that the designers didn’t have in mind when cobbling together little ol’ NDIS.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of James Harper
Sent: Saturday, August 14, 2010 3:49 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] ndis signalling that underlying hardware has changed

Under Xen it is possible to ‘live migrate’ a virtual machine from one physical machine to another. On startup, I get info about the acceleration features that my virtual network card supports - scatter gather, checksum offload, and large send offload. When doing a live migration, it is possible that the new physical machine supports a different feature set than the old physical machine. If this is the case, how should I tell NDIS? The last resort would be to simulate an unplug and replug of the device so that Windows will start it from the start, but maybe there is a better way? I wouldn’t be surprised if there isn’t though as this isn’t a situation that the designers would have had in mind when NDIS was put together…

The device unplug and replug is not really desirable as apart from a slight performance hit while the migration is happening and then a pause of a few milliseconds while the transfer is committed, the user isn’t supposed to be able to tell that this has happened. A difference in network acceleration features is going to be uncommon though.

Thanks

James


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

>

This can be done with an NDIS6 miniport.

I’m using NDIS5 because NDIS6 brings nothing of value to me that’s worth
maintaining two codebases for. This may be one thing though.

When initializing your adapter, claim hardware capabilities for every
offload
that your driver can perform, even if the *current* underlying
physical host
can’t do them.

[NdisMSetMiniportAttributes(NDIS_MINIPORT_ADAPTER_OFFLOAD_ATTRIBUTES.Har
dwareO

ffloadCapabilities)]. Reflect the *current* host capabilities in
DefaultOffloadConfiguration.

Then, as your underlying host changes (and offload capabilities
change),
indicate up a NDIS_STATUS_TASK_OFFLOAD_CURRENT_CONFIG via
NdisMIndicateStatusEx with the new checksum offload and large send
offload
capabilities. TCPIP will react to it accordingly.

Nice.

If you think about it for a moment, you will probably notice that
there is
necessarily a race window where you have just finished migrating to a
new host
that does not support some offload, and your status indication hasn’t
gone up
yet. This is ok – just indicate receive packets without any offload
performed, and TCPIP will fall back to computing checksums on the CPU.
You
can temporarily fail send packets if they have an incompatible
offload.
(Protocols do have to cope with a few dropped packets now and then,
and VM
migration is hopefully an infrequent trauma). If the migration is
slow-ish,
you might want to consider popping a link state indication with
mediadisconnect for the duration, so that overlying protocols
understand that
things are spotty.

Yep. Receive packets are fine, and sent packets I can drop assuming I
don’t drop too many.

I’m not quite sure what you mean w.r.t. scatter-gather changing – I
don’t see
how details of the SG list mapping really affects a guest in a VM. At
that
point, maybe you should just eject the device and re-enumerate it,
since that
is likely something that the designers didn’t have in mind when
cobbling
together little ol’ NDIS.

I pass the list of buffers to Linux. If Linux can only manage a single
page then I have to react accordingly. No sg also limits me to 4096 byte
packet size, eg no 64k large send and no 9k jumbo frames. In actual fact
it’s only a solaris backend that can’t handle sg, so only windows
running on solaris has this problem.

Thanks for the info!

James

I don’t know enough about NDIS5 to claim authoritatively that it does not allow a miniport to change offload capabilities at runtime… but I did poke around some code a bit, and didn’t see anything interesting. Back at the ranch, we’re big fans of NDIS6.

If Linux can only manage a single page then I have to react accordingly. No sg also limits me to 4096 byte packet size, eg no 64k large send and no 9k jumbo frames.

I see. If you are in that situation, you can always start coalescing the packet to a bounce buffer yourself.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of James Harper
Sent: Saturday, August 14, 2010 5:48 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] ndis signalling that underlying hardware has changed

This can be done with an NDIS6 miniport.

I’m using NDIS5 because NDIS6 brings nothing of value to me that’s worth maintaining two codebases for. This may be one thing though.

When initializing your adapter, claim hardware capabilities for every
offload
that your driver can perform, even if the *current* underlying
physical host
can’t do them.

[NdisMSetMiniportAttributes(NDIS_MINIPORT_ADAPTER_OFFLOAD_ATTRIBUTES.Har
dwareO

ffloadCapabilities)]. Reflect the *current* host capabilities in
DefaultOffloadConfiguration.

Then, as your underlying host changes (and offload capabilities
change),
indicate up a NDIS_STATUS_TASK_OFFLOAD_CURRENT_CONFIG via
NdisMIndicateStatusEx with the new checksum offload and large send
offload
capabilities. TCPIP will react to it accordingly.

Nice.

If you think about it for a moment, you will probably notice that
there is
necessarily a race window where you have just finished migrating to a
new host
that does not support some offload, and your status indication hasn’t
gone up
yet. This is ok – just indicate receive packets without any offload
performed, and TCPIP will fall back to computing checksums on the CPU.
You
can temporarily fail send packets if they have an incompatible
offload.
(Protocols do have to cope with a few dropped packets now and then,
and VM
migration is hopefully an infrequent trauma). If the migration is
slow-ish,
you might want to consider popping a link state indication with
mediadisconnect for the duration, so that overlying protocols
understand that
things are spotty.

Yep. Receive packets are fine, and sent packets I can drop assuming I don’t drop too many.

I’m not quite sure what you mean w.r.t. scatter-gather changing – I
don’t see
how details of the SG list mapping really affects a guest in a VM. At
that
point, maybe you should just eject the device and re-enumerate it,
since that
is likely something that the designers didn’t have in mind when
cobbling
together little ol’ NDIS.

I pass the list of buffers to Linux. If Linux can only manage a single page then I have to react accordingly. No sg also limits me to 4096 byte packet size, eg no 64k large send and no 9k jumbo frames. In actual fact it’s only a solaris backend that can’t handle sg, so only windows running on solaris has this problem.

Thanks for the info!

James


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

>

I don’t know enough about NDIS5 to claim authoritatively that it does
not
allow a miniport to change offload capabilities at runtime… but I
did poke
around some code a bit, and didn’t see anything interesting. Back at
the
ranch, we’re big fans of NDIS6.

I figured that NDIS5 wasn’t exactly a high focus piece of code.

> If Linux can only manage a single page then I have to react
accordingly. No
> sg also limits me to 4096 byte packet size, eg no 64k large send and
no 9k
> jumbo frames.

I see. If you are in that situation, you can always start coalescing
the
packet to a bounce buffer yourself.

That’s what I do now when the situation arises. Might be an easier
problem to solve with documentation than code :slight_smile:

James

Having an NDIS5 driver as the only solution (both NT5 and NT6 platforms) is
‘good enough’ if featureless NDIS5 is good enough (like supporting Win2K/XP
guests and low-perf NT6 guests). But I suspect that treating NT5 and NT6 as
two completely different platforms and building a solution for each is a far
better approach in the long run.

The ‘competition’ is NT6 only and perhaps that is an advantage you do not
want to let them have or claim :slight_smile:

Good Luck,
Dave Cattley

>

Having an NDIS5 driver as the only solution (both NT5 and NT6
platforms) is
‘good enough’ if featureless NDIS5 is good enough (like supporting
Win2K/XP
guests and low-perf NT6 guests). But I suspect that treating NT5 and
NT6 as
two completely different platforms and building a solution for each is
a far
better approach in the long run.

The ‘competition’ is NT6 only and perhaps that is an advantage you do
not
want to let them have or claim :slight_smile:

There are a couple of fundamental architecture changes I want to make to
my drivers and once that’s done I can probably put the ndis5 and
scsiport drivers into ‘maintenance only’ mode and create ndis6 and
storport drivers. I can’t imagine that the ndis6 conversion would
involve too much effort beyond search and replace, and last time I
tested storport it was only a few hours effort to convert it to a usable
state… dump and hibernate probably involves a bit more work of course.

I would have stuck with the storport driver but it was for scsi
passthrough (for tape drives etc) and I ran into too many bugs. I was
trying to use a hardware storport driver when I should have been using a
virtual storport driver so I was probably not helping myself there!

Thanks

James