Hello. Total newbie here when it comes to Windows Driver development.
I have been tasked with developing a Windows driver to let a user-land application control/monitor registers on a custom FPGA with a PCIe interface. I made the simplest FPGA firmware load I could think of: Whenever the host writes to the PCIe BAR address, that maps to a hardware register that turns an LED on and off. That’s all it does right now. And it’s single read/writes (no DMA…yet).
I tested this hardware on a Linux machine, and it works. I can toggle the LED. On the Windows machine, I can see the PCIe device enumerated and I can see the BAR values.
I’ve started down the path of Windows Driver development, and taken a few baby steps (Installed Visual Studio stuff: WDK, SDK, etc), got the WinDbg host/target setup going. I used the Visual Studio example project of a KMDF “sample” class driver that pretty much doesn’t do anything other than the DriverEntry and cleanup. For a day, I was able to install the driver, go to the device manager, update the driver of the FPGA card with my “sample” driver, and see a print statement appear in WinDbg, so a little bit of success.
I then started trying to figure out how to expose an “interface” or “API” (if either of those is the correct term), so the user land application can ask the driver to toggle the LED. I fumbled around enough to where now I can no longer find my “sample” driver when in the device manager (it says “operation completed successfully” when I install the inf file, but then it’s nowhere in the list of drivers…). I tried to regress, created a fresh KMDF “sample” class driver from the Visual Studio template…and still no luck.
So I thought I’d take a step back and ask for recommendations for an example driver project that somewhat matches what I’m trying to do.
Stray through the PCI sample driver at https://github.com/microsoft/Windows-driver-samples and watch for the user defined ioctl and skip, for now, all the DMA stuff. Just reading/writing registers on your FPGA from user application is not so difficult.
I then started trying to figure out how to expose an “interface” or “API” (if either of those is the correct term), so the user land application can ask the driver to toggle the LED.
The interface is DeviceIoControl. You can add your own user-mode DLL wrapper to give it a friendly face, but all communication between user mode and kernel mode is done via ReadFile, WriteFile, and DeviceIoControl.
Awesome thank you. I have now compiled and installed the Pci9x5x driver sample, and I can attach it to my FPGA card. I am getting “This device cannot start. (Code 10)”, and “No PLX devices are present and enabled in the system” but I’m not too surprised. My guess is that the driver is trying to talk to a real Pci9x5x device, including all the configuring registers and DMA channels that don’t actually exist on my FPGA card. Does that sound correct?
I really dislike the plx9x5x example driver. It’s based on some very old OSR code, that made sense in something like 2001.
Start from something like the venerable toaster driver… or even just the template that gets generated by VS. Then add Prepare Hardware and Device Control callbacks. That’s all you need to do.
That’ll keep you away from the mess that’s in the PLX9x5x driver, which is way more complexity than you need.
I have been tasked with developing a Windows driver to let a user-land application control/monitor registers on a custom FPGA with a PCIe interface.
I would be remiss if I did not comment on this as well.
In general, just writing a driver that maps a user-land app into the I/O space described by a BAR in a driver is not a good idea. You’re taking a privileged activity (talking to hardware) that could affect the overall stability of the system and exposing that to the non-privileged world. This is ripe for abuse, both intentional and unintentional.
When we do this type of thing for custom FPGAs, we generally work very hard at separating the FPGA registers that manage the device’s application tasks does from those registers that control the device itself and can affect the overall stability of the system. We put those two things in separate BARs. We may map the “non-critical, application oriented” stuff back to user space… to let the app have direct control over FPGA parameters and application-type functions. And the driver exclusively maps and uses the other BAR to control the device (reset, setup DMA operations, or whatever). Separating the two spaces is pretty easy from the FPGA side (especially if you think about it from the beginning).
Also, you need to note that mapping I/O space of the device back to user-land involves more complexity and security issues than might initially be evident. Lots can go wrong… and in general mapping stuff from kernel mode back into user-space is not considered a “best practice” at all.
Finally, please note that I’m talking here about systems dedicated to a particular purpose (process control, medical systems, etc). For general purpose systems, it would rarely be a good idea to allow user-mode any direct access to the hardware registers at all.
Peter makes a very good point here. When I’ve done drivers for custom devices, I’ve always striven to make the interface an abstraction. The user-mode code should specify what it wants, and not how it should be done. That not only adds safety, but it often gives them the opportunity to move to a next generation design without having to rewrite the application code.
I haven’t always succeeded in that. A couple of clients really, really wanted to drive the FPGA registers directly. That’s OK, but I always leave the DMA mapping to the driver. User mode code cannot be trusted with physical addresses.
Understood on not wanting to give the keys to the kingdom to user-space. So I shouldn’t expose addresses/registers directly to user space, only the driver will perform those transfers. The user space application will only make higher level/abstracted requests to the driver (“driver, will you turn on the LED?”) and the driver takes care of the sequencing/control/register/memory accesses appropriately to achieve the goal.
Yet another voice to the choir … there are literally no FPGA registers which I expose to usermode, everything is through a driver. Adding to the excellent reasons mentioned above, there is another reason you really want to abstract the FPGA registers: hardware updates. In FPGA land most hardware seems to change if not weekly then at least monthly as new FPGA’s supplant old ones.
The design pattern which I’ve found gives me the most flexibility is to think of the driver as having three “edges”; a usermode edge (where the IOCTL’s and file I/O handlers come in, an OS edge (where the various PnP notifications come in) and a physical edge (which is where the driver talks to the hardware). Each of these edges is abstracted; there is a base class with general functions, and a concrete layer instantiated at runtime which is where the actual functions are implemented. By doing it this way you can support multiple IOCTL interfaces (needed for the security and functionality reasons mentioned) at the usermode edge, different OS flavors (Win10 and it’s mutations, and now Win11) at the OS edge and different hardware chipsets (the FPGA flavor of the month club) at the hardware edge.
The OS edge is chosen at DriverEntry based on driver version info, the usermode edge is chosen at CreateDevice based on registry values, and the hardware edge is chosen in PrepareHardware when the hardware is loaded and then it’s just moving data …
@“Peter_Viscarola_(OSR)” separation of data versus command/ control BAR’s is definitely something to consider if you have that ability, I’ll definitely be doing that with my FPGA projects moving forward …
Another point to consider is that if device resources are mapped into a UM process, there is no opportunity for multiple UM processes to interact with it at the same time. That may not be important, but keeping the resources within KM does allow a driver to allow multiple processes to interact with a device simultaneously - and it provides the rules as to how those interactions can happen.
