PCIe hot plug

Hi everyone,
We have a PCIe based device and a WDF based driver that supports the device. We would like to add hot plug/hot remove capability to the driver.

I was looking for general guide lines on how to add hot plug/remove support to the driver?
What capabilities KMDF has to support hot plug for PCIe based devices?

Thanks for your help
pada

Anything related to the plugging in of the device will have to be handled by the bus driver (pci.sys). can’t speak to what support is there though. What OS are you targeting? As for hot removal, again the bus has to support it to report the device as missing, but from a KMDF driver perspective, it is just like power down. KMDF fully supports surprise removal of a device.

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@hotmail.com
Sent: Friday, December 16, 2011 1:33 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] PCIe hot plug

Hi everyone,
We have a PCIe based device and a WDF based driver that supports the device. We would like to add hot plug/hot remove capability to the driver.

I was looking for general guide lines on how to add hot plug/remove support to the driver?
What capabilities KMDF has to support hot plug for PCIe based devices?

Thanks for your help
pada


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Doron, thanks for your feedback.
My OS is 2008 R1 and R2.

If I reboot my system without my card being installed, my DriverEntry() function gets called but my EvtDeviceAdd() does not gets called. Inserting my device after a reboot will not trigger the EvtDeviceAdd(). Since EvtDeviceAdd() is not called no device is created and my pnp callbacks are not initialized.

What needs to be done here to detect that a new device is inserted?

Thanks again
pada

Uh ? fix your INF file so that the vendor and device IDs identify your driver package?

Gary Little
H (952) 223-1349
C (952) 454-4629
xxxxx@comcast.net

On Dec 16, 2011, at 5:01 PM, xxxxx@hotmail.com wrote:

Doron, thanks for your feedback.
My OS is 2008 R1 and R2.

If I reboot my system without my card being installed, my DriverEntry() function gets called but my EvtDeviceAdd() does not gets called. Inserting my device after a reboot will not trigger the EvtDeviceAdd(). Since EvtDeviceAdd() is not called no device is created and my pnp callbacks are not initialized.

What needs to be done here to detect that a new device is inserted?

Thanks again
pada


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Gary,
My INF file has the correct VendorId and deviceId. Otherwise, EvtDeviceAdd() wouldn’t get called after a reboot with device installed.

My issue is EvtDeviceAdd() does not get called if I install my card after reboot.

Thanks

wrote in message news:xxxxx@ntdev…
> Gary,
> My INF file has the correct VendorId and deviceId. Otherwise,
> EvtDeviceAdd() wouldn’t get called after a reboot with device installed.
>
> My issue is EvtDeviceAdd() does not get called if I install my card after
> reboot.
>

Hot-pluggable PCIe requires special hardware support (maybe in BIOS as
well). Does your machine have it?
From what you describe it looks like the BIOS caches stale PCI config data.
– pa

Look in device manager after you hot plug the card, does it show up in the device tree? If not, your machine doesn’t support it

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Pavel A.
Sent: Friday, December 16, 2011 3:28 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] PCIe hot plug

wrote in message news:xxxxx@ntdev…
> Gary,
> My INF file has the correct VendorId and deviceId. Otherwise,
> EvtDeviceAdd() wouldn’t get called after a reboot with device installed.
>
> My issue is EvtDeviceAdd() does not get called if I install my card
> after reboot.
>

Hot-pluggable PCIe requires special hardware support (maybe in BIOS as well). Does your machine have it?
From what you describe it looks like the BIOS caches stale PCI config data.
– pa


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Device manager does not show my device unless I do “Scan for HW changes”. ( my EvtDeviceAdd() gets also called).
However it shows the device with yellow exclamation mark. Under the device status it says that
“The device can not find enough free resources that it can use. Code 12”

Does it mean that OS did not do the enumeration after I inserted my device?

It means that the pci driver did not auto detect the insertion, but when it was told to go manual rescan, it found your device. The code 12 could very well be that the device was found, but the rest of the devices were not rebalanced to allow for the requested resources to be assigned.

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@hotmail.com
Sent: Friday, December 16, 2011 4:29 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] PCIe hot plug

Device manager does not show my device unless I do “Scan for HW changes”. ( my EvtDeviceAdd() gets also called).
However it shows the device with yellow exclamation mark. Under the device status it says that “The device can not find enough free resources that it can use. Code 12”

Does it mean that OS did not do the enumeration after I inserted my device?


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

If your device can operate without a ROM that runs as part of BIOS or UEFI
startup, then you’ll probably already support hot-plug. If you’re using
KMDF, then you’d have to go out of your way to break hot-plug support.

Supporting clean surprise-removal (where the user just yanks out the device
without cleanly requesting that it be ejected) is harder. You have to
design the hardware so that there are no registers where you can’t tell the
difference between the device disappearing and valid reads. Once you’ve
done that, you need to structure your code so that no garbage reads from the
device will cause your code to do unbounded things.

Devices tend to exist along a spectrum, from the simplest device where all
control happens via programmed I/O to registers on the device to a device
where all interaction with the device, after initial setup, happens by
reading and writing work lists in memory, usually through a common buffer.
Initiating I/O then involves setting up a work list and then writing to a
“doorbell” register on the device.

Devices at the simpler end of the spectrum are harder to prove correct in
this respect when the device might disappear between any two instructions in
your driver.

Beyond the hardware considerations, supporting hot-plug well requires that
the upper-edge interface to your driver is well designed. (You don’t say
what it is, and that would narrow the topic considerably.) When your device
is unplugged, there will presumably be handles open to it. The device
instance won’t be removed until all of these handles are closed. Your
client applications need to listen for PnP notifications, or you have to
help them along by pending an IOCTL that gets completed when they should
tear down and close their handles. (The former is general, and works for
all drivers, the latter is less complicated.)

It’s entirely likely that you will eventually encounter a situation where
your device is removed and then reinserted, faster than the teardown for the
first instance can complete. For this period of time, it will be the case
that there are two instances of your driver active for the very same device.
To be sure that this isn’t a problem, you need to stop initiating attempts
to access your device the very moment that you know that it’s gone. And you
absolutely must guarantee that no further hardware access occurs after you
complete IRP_MN_SURPRISE_REMOVAL. As soon as that IRP completes, the PnP
manager may load up the second instance of your driver. You don’t want the
two contending for your device.

Jake Oshins
Windows Kernel Group

This post implies no warranties and confers no rights.


wrote in message news:xxxxx@ntdev…

Hi everyone,
We have a PCIe based device and a WDF based driver that supports the device.
We would like to add hot plug/hot remove capability to the driver.

I was looking for general guide lines on how to add hot plug/remove support
to the driver?
What capabilities KMDF has to support hot plug for PCIe based devices?

Thanks for your help
pada

Supporting PCIe hot plug requires very specific behavior from the
physical device and the PCIe root port that the device is inserted
into. The root port must support hot plug (also known as surprise
insertion). Most of the server grade chipsets support surprise
insertion but your PCIe end point on your device MUST adhere to PCI
requirements for PCIe surprise insertion. See the PCI specs for
detailed information. Did your hardware team give you detailed
information regarding the devices support for surprise insertion? If
not, request it. This is non-trivail stuff to get right…

On Sun, Dec 18, 2011 at 12:07 AM, Jake Oshins
wrote:
> If your device can operate without a ROM that runs as part of BIOS or UEFI
> startup, then you’ll probably already support hot-plug. ?If you’re using
> KMDF, then you’d have to go out of your way to break hot-plug support.
>
> Supporting clean surprise-removal (where the user just yanks out the device
> without cleanly requesting that it be ejected) is harder. ?You have to
> design the hardware so that there are no registers where you can’t tell the
> difference between the device disappearing and valid reads. ?Once you’ve
> done that, you need to structure your code so that no garbage reads from the
> device will cause your code to do unbounded things.
>
> Devices tend to exist along a spectrum, from the simplest device where all
> control happens via programmed I/O to registers on the device to a device
> where all interaction with the device, after initial setup, happens by
> reading and writing work lists in memory, usually through a common buffer.
> Initiating I/O then involves setting up a work list and then writing to a
> “doorbell” register on the device.
>
> Devices at the simpler end of the spectrum are harder to prove correct in
> this respect when the device might disappear between any two instructions in
> your driver.
>
> Beyond the hardware considerations, supporting hot-plug well requires that
> the upper-edge interface to your driver is well designed. ?(You don’t say
> what it is, and that would narrow the topic considerably.) ?When your device
> is unplugged, there will presumably be handles open to it. ?The device
> instance won’t be removed until all of these handles are closed. ?Your
> client applications need to listen for PnP notifications, or you have to
> help them along by pending an IOCTL that gets completed when they should
> tear down and close their handles. ?(The former is general, and works for
> all drivers, the latter is less complicated.)
>
> It’s entirely likely that you will eventually encounter a situation where
> your device is removed and then reinserted, faster than the teardown for the
> first instance can complete. ?For this period of time, it will be the case
> that there are two instances of your driver active for the very same device.
> To be sure that this isn’t a problem, you need to stop initiating attempts
> to access your device the very moment that you know that it’s gone. ?And you
> absolutely must guarantee that no further hardware access occurs after you
> complete IRP_MN_SURPRISE_REMOVAL. ?As soon as that IRP completes, the PnP
> manager may load up the second instance of your driver. ?You don’t want the
> two contending for your device.
>
> Jake Oshins
> Windows Kernel Group
>
> This post implies no warranties and confers no rights.
>
> --------------------------------------------------------------
> wrote in message news:xxxxx@ntdev…
>
>
> Hi everyone,
> We have a PCIe based device and a WDF based driver that supports the device.
> We would like to add hot plug/hot remove capability to the driver.
>
> I was looking for general guide lines on how to add hot plug/remove support
> to the driver?
> What capabilities ?KMDF has to support hot plug for PCIe based devices?
>
> Thanks for your help
> pada
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer


Robert Randall | xxxxx@gmail.com

xxxxx@hotmail.com wrote:

My INF file has the correct VendorId and deviceId. Otherwise, EvtDeviceAdd() wouldn’t get called after a reboot with device installed.

Well, there is some mistake in your installation, because your
DriverEntry should not be called until immediately before the first
EvtDeviceAdd. If DriverEntry is getting called at boot time, before any
hardware has been detected, then you have a problem.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Guys, Thank you very much for the guidelines.
We are still in the learning and investigation mode, so I appreciate all your advices.

Regarding to the issue of “Device Manager does not show my device unless I do “Scan for HW” then it shows the device with yellow exclamation mark”, is
this a limitation on my system or my device? If not, what needs to be done that OS understand the arrival of the new device?

Also on the surprise removal, my EvtDeviceSurpriseRemoval() does not get called until I manually run a “Scan for new HW” in the Device Manager. I have SurpriseRemovalOK = WdfTrue
Is this again a limitation on my system or my device?

Thanks again.

Yes. It’s worth noting that I answered the question entirely from the point
of view of your driver software, not the hardware. Sorry.

Jake Oshins
Windows Kernel Group

This post implies no warranties and confers no rights.


“Robert Randall” wrote in message news:xxxxx@ntdev…

Supporting PCIe hot plug requires very specific behavior from the
physical device and the PCIe root port that the device is inserted
into. The root port must support hot plug (also known as surprise
insertion). Most of the server grade chipsets support surprise
insertion but your PCIe end point on your device MUST adhere to PCI
requirements for PCIe surprise insertion. See the PCI specs for
detailed information. Did your hardware team give you detailed
information regarding the devices support for surprise insertion? If
not, request it. This is non-trivail stuff to get right…

On Sun, Dec 18, 2011 at 12:07 AM, Jake Oshins
wrote:
> If your device can operate without a ROM that runs as part of BIOS or UEFI
> startup, then you’ll probably already support hot-plug. If you’re using
> KMDF, then you’d have to go out of your way to break hot-plug support.
>
> Supporting clean surprise-removal (where the user just yanks out the
> device
> without cleanly requesting that it be ejected) is harder. You have to
> design the hardware so that there are no registers where you can’t tell
> the
> difference between the device disappearing and valid reads. Once you’ve
> done that, you need to structure your code so that no garbage reads from
> the
> device will cause your code to do unbounded things.
>
> Devices tend to exist along a spectrum, from the simplest device where all
> control happens via programmed I/O to registers on the device to a device
> where all interaction with the device, after initial setup, happens by
> reading and writing work lists in memory, usually through a common buffer.
> Initiating I/O then involves setting up a work list and then writing to a
> “doorbell” register on the device.
>
> Devices at the simpler end of the spectrum are harder to prove correct in
> this respect when the device might disappear between any two instructions
> in
> your driver.
>
> Beyond the hardware considerations, supporting hot-plug well requires that
> the upper-edge interface to your driver is well designed. (You don’t say
> what it is, and that would narrow the topic considerably.) When your
> device
> is unplugged, there will presumably be handles open to it. The device
> instance won’t be removed until all of these handles are closed. Your
> client applications need to listen for PnP notifications, or you have to
> help them along by pending an IOCTL that gets completed when they should
> tear down and close their handles. (The former is general, and works for
> all drivers, the latter is less complicated.)
>
> It’s entirely likely that you will eventually encounter a situation where
> your device is removed and then reinserted, faster than the teardown for
> the
> first instance can complete. For this period of time, it will be the case
> that there are two instances of your driver active for the very same
> device.
> To be sure that this isn’t a problem, you need to stop initiating attempts
> to access your device the very moment that you know that it’s gone. And
> you
> absolutely must guarantee that no further hardware access occurs after you
> complete IRP_MN_SURPRISE_REMOVAL. As soon as that IRP completes, the PnP
> manager may load up the second instance of your driver. You don’t want
> the
> two contending for your device.
>
> Jake Oshins
> Windows Kernel Group
>
> This post implies no warranties and confers no rights.
>
> --------------------------------------------------------------
> wrote in message news:xxxxx@ntdev…
>
>
> Hi everyone,
> We have a PCIe based device and a WDF based driver that supports the
> device.
> We would like to add hot plug/hot remove capability to the driver.
>
> I was looking for general guide lines on how to add hot plug/remove
> support
> to the driver?
> What capabilities KMDF has to support hot plug for PCIe based devices?
>
> Thanks for your help
> pada
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer


Robert Randall | xxxxx@gmail.com