We have developed a PCIe card that streams raw video over a 10G Ethernet interface.
The video contents to be streamed is provided by another board, a GPU.
Our board exposes 3 BARs in its PCI config space. The board driver is WDF based.
At startup, the GPU application fetches the physical address of BAR2 and of one other “flush” register from our board via an IOCTL.
It then DMAs video content into the BAR2 memory and writes to this “flush” register that triggers our HW network transmission.
Although unconventional, this all works fine but the problems arise at shutdown of the PC:
- our driver will unmap the BAR2 memory upon shutdown callback
- unaware of this, the GPU may asynchronously still DMA a frame into this memory
- result is a BSOD - pointing to a level2 cache incoherency
This happens randomly, about 1/100 times and it all seems to make sense, since the GPU application/driver will most probably be shut down later than our PCIe driver.
Is there any regular way to alert the GPU application of our driver being shut down?
e.g. by sending an event to the user application ?
Or is there a way to make sure that our driver will be unloaded later than the GPU ?
Thank you in advance for reading this.
- Bernard Willaert
Barco, Belgium
Healthcare Division
> Is there any regular way to alert the GPU application of our driver being shut down?
On Win7+: MN_DEVICE_RELATIONS/PowerRelations
–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com
Thanks for your answer, Maxim.
Our OS is Win7, but MN_DEVICE_RELATIONS/PowerRelations is still unclear to me ?
Regards,
Applications will see power messages, but that is not synchronized with your specific device power state. You could have the application pend an IOCtl which you complete during power down. The completion tells the app to stop, but that isn’t quite enough as you don’t know how long it will take the app to stop, so you need a way for the app to indicate success. The simplest way is for the app to close its handle to your driver. If your block the power down irp (and thus delay the unmap) until the close handle you should be fine. The risk is that it takes the application too long to close the handle and you get a 9f power timeout but check.
Sent from Outlook Mailhttp: for Windows 10 phone
From: xxxxx@barco.com
Sent: Friday, October 23, 2015 11:20 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Driver shutdown synchronization
We have developed a PCIe card that streams raw video over a 10G Ethernet interface.
The video contents to be streamed is provided by another board, a GPU.
Our board exposes 3 BARs in its PCI config space. The board driver is WDF based.
At startup, the GPU application fetches the physical address of BAR2 and of one other “flush” register from our board via an IOCTL.
It then DMAs video content into the BAR2 memory and writes to this “flush” register that triggers our HW network transmission.
Although unconventional, this all works fine but the problems arise at shutdown of the PC:
- our driver will unmap the BAR2 memory upon shutdown callback
- unaware of this, the GPU may asynchronously still DMA a frame into this memory
- result is a BSOD - pointing to a level2 cache incoherency
This happens randomly, about 1/100 times and it all seems to make sense, since the GPU application/driver will most probably be shut down later than our PCIe driver.
Is there any regular way to alert the GPU application of our driver being shut down?
e.g. by sending an event to the user application ?
Or is there a way to make sure that our driver will be unloaded later than the GPU ?
Thank you in advance for reading this.
- Bernard Willaert
Barco, Belgium
Healthcare Division
—
NTDEV is sponsored by OSR
Visit the list at: https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.osronline.com%2Fshowlists.cfm%3Flist%3Dntdev&data=01|01|Doron.Holan%40microsoft.com|4827f3e35b9d42541c7b08d2dc3b2b24|72f988bf86f141af91ab2d7cd011db47|1&sdata=9hUKjw6nU75rsN3Hik%2FAYbG1oJENv3azLMMJGe1PY%2Bo%3D
OSR is HIRING!! See https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.osr.com%2Fcareers&data=01|01|Doron.Holan%40microsoft.com|4827f3e35b9d42541c7b08d2dc3b2b24|72f988bf86f141af91ab2d7cd011db47|1&sdata=mCde%2FLSa6BJj2ARjhZW7GEw%2BbAjtFsJJ8tl%2FBexvslo%3D
For our schedule of WDF, WDM, debugging and other seminars visit:
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.osr.com%2Fseminars&data=01|01|Doron.Holan%40microsoft.com|4827f3e35b9d42541c7b08d2dc3b2b24|72f988bf86f141af91ab2d7cd011db47|1&sdata=cplj91qnPyJ7zP6F7%2Bj7I0%2FQPLaIwF1vmhs%2FdxbDA9A%3D
To unsubscribe, visit the List Server section of OSR Online at https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.osronline.com%2Fpage.cfm%3Fname%3DListServer&data=01|01|Doron.Holan%40microsoft.com|4827f3e35b9d42541c7b08d2dc3b2b24|72f988bf86f141af91ab2d7cd011db47|1&sdata=X1HxIxgXbbTmo3Jwet2btU%2BESlxu3cQlnVyTK%2BXYO%2Bk%3D</http:>
Thanks, Doron for your answer!
You could have the application pend an IOCtl which you complete during power down.
This is also a solution we were considering, that is well within the reach of our knowledge.
We complete the IOCTL within the power down callback of the PCIe driver and then delay the unmap for e.g. another 50 ms. This should be more than sufficient for the last DMA transaction to complete before the memory is unmapped. For the GPU application this would be the sign to “shut up” and no longer DMA anything to our memory.
Thanks again !
You still have a race between the app processing the completion and the unmapping of the bars. You need an ACK of some kind from the app that is has stopped touching the hardware.
Sent from Outlook Mailhttp: for Windows 10 phone
From: xxxxx@barco.com
Sent: Saturday, October 24, 2015 8:43 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Driver shutdown synchronization
Thanks, Doron for your answer!
>You could have the application pend an IOCtl which you complete during power down.
This is also a solution we were considering, that is well within the reach of our knowledge.
We complete the IOCTL within the power down callback of the PCIe driver and then delay the unmap for e.g. another 50 ms. This should be more than sufficient for the last DMA transaction to complete before the memory is unmapped. For the GPU application this would be the sign to “shut up” and no longer DMA anything to our memory.
Thanks again !
- Bernard
—
NTDEV is sponsored by OSR
Visit the list at: https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.osronline.com%2Fshowlists.cfm%3Flist%3Dntdev&data=01|01|Doron.Holan%40microsoft.com|b0e5facd66ab47c9692608d2dc89e880|72f988bf86f141af91ab2d7cd011db47|1&sdata=9jip1YlwAa6ERMZ5s8q4u7OCU0YYrkIJ2lmhnZFTT%2Bg%3D
OSR is HIRING!! See https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.osr.com%2Fcareers&data=01|01|Doron.Holan%40microsoft.com|b0e5facd66ab47c9692608d2dc89e880|72f988bf86f141af91ab2d7cd011db47|1&sdata=7Bp%2FfvB9LAzzrHbUdc4p%2BDIVQmZvjPvYUIPrEUZ%2BIW4%3D
For our schedule of WDF, WDM, debugging and other seminars visit:
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.osr.com%2Fseminars&data=01|01|Doron.Holan%40microsoft.com|b0e5facd66ab47c9692608d2dc89e880|72f988bf86f141af91ab2d7cd011db47|1&sdata=ePHT19Of3an%2BFaTvNmTL7fVyDNUTKfhkrAoQnYu1QQ4%3D
To unsubscribe, visit the List Server section of OSR Online at https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.osronline.com%2Fpage.cfm%3Fname%3DListServer&data=01|01|Doron.Holan%40microsoft.com|b0e5facd66ab47c9692608d2dc89e880|72f988bf86f141af91ab2d7cd011db47|1&sdata=GvEoaVYhhq%2BLAg6HkE9EVRKPXrV7aVhyCU01ie%2B4sow%3D</http:>