Auto-reconfigurable PCIe device

Hello all,
I read a lot of things on the subject, but I wasn’t able to apply those readings to my problem.
I have a PCIe device that is made of an FPGA that contains a PCIe block.
By writing to a specific register, I can launch a reconfiguration (reprogrammation) of this FPGA from a specific address into a flash memory.
During the reconfiguration, the PCIe link disappears and when the reconfiguration is done, the PCIe link is retrained.
The issue is that my KMDF driver is not accessible anymore after the reconfiguration.
I have to perform a “devcon.exe disable PCI\VEN_xxxx&DEV_xxxx” and the a “devcon.exe enable PCI\VEN_xxxx&DEV_xxxx”.
I would like to make this disable/enable trick automatic when my PCIe device reappears.
I’m totally lost with all “EjectSupported”, “Removable” and other “SurpriseRemovalOK” stuff and can’t figure out how to automate this process.
Could anybody help me on the subject?
Thanks a lot in advance.
Vincent

Your driver has to support SurpriseRemovalOK if the device is going to get
yanked out from under it. Note that not all platforms fully support hot
removal of pci(e) devices, so you may need to use a user mode service to
force a rescan of the pci bus in order to get the old device removed and
the new device added. The devcon sample illustrates how to do this from
user mode.

Mark Roddy

On Mon, Mar 16, 2015 at 9:39 AM, wrote:

> Hello all,
> I read a lot of things on the subject, but I wasn’t able to apply those
> readings to my problem.
> I have a PCIe device that is made of an FPGA that contains a PCIe block.
> By writing to a specific register, I can launch a reconfiguration
> (reprogrammation) of this FPGA from a specific address into a flash memory.
> During the reconfiguration, the PCIe link disappears and when the
> reconfiguration is done, the PCIe link is retrained.
> The issue is that my KMDF driver is not accessible anymore after the
> reconfiguration.
> I have to perform a “devcon.exe disable PCI\VEN_xxxx&DEV_xxxx” and the a
> “devcon.exe enable PCI\VEN_xxxx&DEV_xxxx”.
> I would like to make this disable/enable trick automatic when my PCIe
> device reappears.
> I’m totally lost with all “EjectSupported”, “Removable” and other
> “SurpriseRemovalOK” stuff and can’t figure out how to automate this process.
> Could anybody help me on the subject?
> Thanks a lot in advance.
> Vincent
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

xxxxx@cea.fr wrote:

I have a PCIe device that is made of an FPGA that contains a PCIe block.
By writing to a specific register, I can launch a reconfiguration (reprogrammation) of this FPGA from a specific address into a flash memory.
During the reconfiguration, the PCIe link disappears and when the reconfiguration is done, the PCIe link is retrained.
The issue is that my KMDF driver is not accessible anymore after the reconfiguration.
I have to perform a “devcon.exe disable PCI\VEN_xxxx&DEV_xxxx” and the a “devcon.exe enable PCI\VEN_xxxx&DEV_xxxx”.
I would like to make this disable/enable trick automatic when my PCIe device reappears.

There are actually a number of layers that have to play together to make
this work.

The PCIExpress spec does allow hot-plugging, where a device is either
added or removed on the fly. That requires support in the root complex
chipset and in the BIOS, and not all of them support it. Remember, in
order for your removal and arrival to be seen, some piece of hardware
has to report that “one of my devices has left the building.” The bus
driver for that bus has to be able to notice that, and report the change
to PnP, who will then tear your stack down and build another.

Your existing driver will need to be unloaded, and a new one will
replace it. It this respect, this is just like a USB device undergoing
unplug and replug.

When your device leaves the bus, do you see it being removed from Device
Manager? Does your driver get a “surprise removal” notification? If
not, then you have larger problems then just your driver.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Thanks a lot for your answers.
I think hot-unplugging/plugging is not supported by my hardware.
In fact, I don’t receive any surprise removal notification in my driver.
So, I’ll have to think about a sort of “work-around” solution based on disable/enable of my driver.
The problem I have with this, is that the “SetupDiChangeState” function requires to be called by a member of the administrator group.

“requires to be called by a member of the administrator group” - so you
need to provide a service. Also you aren’t receiving surprise removal
notification because the parent (pci) bus driver is unaware of the state
change. If your pcie device changes its enumerated hardware identifiers
when it reconfigures, and if your service forces a re-enumeration of the
parent pci bus, surprise removal will occur followed by arrival of your
“new” device.

Mark Roddy

On Tue, Mar 17, 2015 at 4:13 AM, wrote:

> Thanks a lot for your answers.
> I think hot-unplugging/plugging is not supported by my hardware.
> In fact, I don’t receive any surprise removal notification in my driver.
> So, I’ll have to think about a sort of “work-around” solution based on
> disable/enable of my driver.
> The problem I have with this, is that the “SetupDiChangeState” function
> requires to be called by a member of the administrator group.
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

Hi Mark,
After issuing a write to the register to force the reconfiguration of my PCIe device, I called the “WdfDeviceSetFailed” function along with the “WdfDeviceFailedAttemptRestart” parameter, just to see.
The following callbacks get called inside my driver, and in the following order :

  1. EvtDeviceSurpriseRemoval
  2. EvtDeviceD0Exit
  3. EvtDeviceReleaseHardware
  4. EvtCleanupCallback
    However, no trace of a restart.
    The driver is marked with a yellow exclamation point, and that’s all.
    What do you mean when you write “If your pcie device changes its enumerated hardware identifiers
    when it reconfigures”, what kind of hardware identifier can be changed like this?
    As I write the PCIe device code myself, it would be an option to change its behaviour.
    Vincent

WdfDeviceFailedAttemptRestart requires the underlying bus driver (PCI) to support reenemeration of the PDO it created. The version of PCI you are running on does not support reenum (I think it was added in win8.1).

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@cea.fr
Sent: Tuesday, March 17, 2015 8:07 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Auto-reconfigurable PCIe device

Hi Mark,
After issuing a write to the register to force the reconfiguration of my PCIe device, I called the “WdfDeviceSetFailed” function along with the “WdfDeviceFailedAttemptRestart” parameter, just to see.
The following callbacks get called inside my driver, and in the following order :

  1. EvtDeviceSurpriseRemoval
  2. EvtDeviceD0Exit
  3. EvtDeviceReleaseHardware
  4. EvtCleanupCallback
    However, no trace of a restart.
    The driver is marked with a yellow exclamation point, and that’s all.
    What do you mean when you write “If your pcie device changes its enumerated hardware identifiers when it reconfigures”, what kind of hardware identifier can be changed like this?
    As I write the PCIe device code myself, it would be an option to change its behaviour.
    Vincent

NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

If your “reconfigured” device can be managed by the same function driver
then you don’t need to change pnp ids. Going back to your original post,
that seems to be the case. As Doron noted you are using functionality that
it seems only works in 8.1 or later. The other way is to implement a
service that performs the disable enable for you. That should work for all
current versions of windows.

Mark Roddy

On Tue, Mar 17, 2015 at 2:10 PM, Doron Holan
wrote:

> WdfDeviceFailedAttemptRestart requires the underlying bus driver (PCI) to
> support reenemeration of the PDO it created. The version of PCI you are
> running on does not support reenum (I think it was added in win8.1).
>
> d
>
> -----Original Message-----
> From: xxxxx@lists.osr.com [mailto:
> xxxxx@lists.osr.com] On Behalf Of
> xxxxx@cea.fr
> Sent: Tuesday, March 17, 2015 8:07 AM
> To: Windows System Software Devs Interest List
> Subject: RE:[ntdev] Auto-reconfigurable PCIe device
>
> Hi Mark,
> After issuing a write to the register to force the reconfiguration of my
> PCIe device, I called the “WdfDeviceSetFailed” function along with the
> “WdfDeviceFailedAttemptRestart” parameter, just to see.
> The following callbacks get called inside my driver, and in the following
> order :
> 1. EvtDeviceSurpriseRemoval
> 2. EvtDeviceD0Exit
> 3. EvtDeviceReleaseHardware
> 4. EvtCleanupCallback
> However, no trace of a restart.
> The driver is marked with a yellow exclamation point, and that’s all.
> What do you mean when you write “If your pcie device changes its
> enumerated hardware identifiers when it reconfigures”, what kind of
> hardware identifier can be changed like this?
> As I write the PCIe device code myself, it would be an option to change
> its behaviour.
> Vincent
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

OK, if the reenumeration of the PDO is not supported by my bus driver, then no other way than making a user mode application restart the driver.
And as admin is required, I must use a Windows service.
Thank you for all those informations.
Do you know of a way to inform a running service that a device disappeared or that a driver was stopped, other than polling? Any PnpManager notification?

This reconfiguration doesn’t happen spontaneously, right? Your function
driver could inform your service that “I’ve reconfigured, please restart
me”. The usual way to do this would be your own IOCTL based API that your
service would use to control the device state.

Mark Roddy

On Wed, Mar 18, 2015 at 4:56 AM, wrote:

> OK, if the reenumeration of the PDO is not supported by my bus driver,
> then no other way than making a user mode application restart the driver.
> And as admin is required, I must use a Windows service.
> Thank you for all those informations.
> Do you know of a way to inform a running service that a device disappeared
> or that a driver was stopped, other than polling? Any PnpManager
> notification?
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>