I am trying to implement an inverted IOCTL model for driver-driver communication. Lets call them driver A and driver B.
Driver B pends a request to Driver A. Driver A waits for notification from user mode and once that happens, it will complete the pended request from Driver B. Once this request is completed, driver B would do necessary action and would pend request to Driver A again.
My question being what happens when the user mode sends a notification to Driver A but Driver B hasn’t pended request to Driver A yet. How to handle such scenarios? I don’t want Driver B to miss any notification from User mode. Note that there is no way for direct communication between Usermode and Driver B(it has to happen via Driver A).
This happens in following case:
When driver B is currently processing completed request from Driver A(i.e it hasn’t pended any request to Driver A yet), and a notification happens from user mode.
To give you a visual, this is the flow I am trying to implement:
usermode->DriverA->DriverB.
Driver B should pend at least 2 IOCTLs, maybe more depending on your load and processing time
Driver A should implement a recording of notifications received from user mode if there are no pended B requests. I have used both ring buffers or double linked lists.
When Driver A processes B’s request, it either completes the request immediately with an entry from the history or queues the request.
When Driver A processes B’s request, it either completes the request immediately with an entry from the history or queues the request.
Can you elaborate on above statement. From what I understand, if Driver B has not pended request to Driver A and Driver A gets notification from user mode, it will queue the request in its internal ring buffer/linked list. Is my understanding correct?
Yes. You can hold onto the notification request from user mode until driver B sends a request to retrieve it. There are two things to account for
the pended application needs to be cancelable
the application needs to handle the condition where the request is pended. if the application is sending synchronous IO, pending the request will block the sending thread. Even if the application is sending async I/O, it must handle the condition that >1 requests can be sent at once and pended
if you can copy the data out of the user mode notification request into an internal data structure you can avoid both issues.
It should be noted that pending 2 IRPs is a minimum. The best number to pend depends on complex factors and may be impossible to determine. When in doubt, pending more is usually never a bad thing.
in general, the less certain you are about the time it will take to process a completion, and the rate at which completions occur, the deeper you want your queue of pending IRPs. It costs extra memory, but usually not very much.
Doron’s points 2 & 3 are designed to handle the case where despite your best efforts, the queue of pending IRPs is empty. If you can lose data, then you can just drop the data at this point. If you can lose data, but want to include a safeguard, then a circular buffer will do that. I prefer sending extra pending IRPs, but there is an argument to be made. If you cannot lose data, then you must keep an expandable list. Make sure to include a circuit breaker / safety valve in this design
How do we pend multiple IRPs, just open the handle multiple times to Driver A and send different overlapped structure to it?
I am trying to look for existing samples on the web but had a difficult time finding it out.
open one handle as overlapped, send multiple IO request on the same handle. You can use the thread pool APIs, see CreateThreadpoolIo, to help manage the requests.
Thankyou. Looks like since this is driver to driver inverted IOCTL model, I need to use WDF based functions.
I plan on pending the request to target driver using the following but have certain question(more on that later):
Format the request using WdfIoTargetFormatRequestForRead.
Register completion routine using WdfRequestSetCompletionRoutine .
Finally send async request using WdfRequestSend.
In the scenario where Driver B has sent two pending requests to Driver A, and Driver A notifies Driver B to complete the first pending request, it triggers the completion routine of Driver B. Now, if Driver A completes another pending request while Driver B is still processing the first completion routine, the question is whether Driver B can invoke the same completion routine once again. Is it possible for Driver B to invoke the completion routine twice(concurrently), or does it happen in a serialized fashion?
driver to driver or UM to driver inverted call doesn’t make a lot of difference. Just use the right APIs, but the ideas are the same
In every case, you want only ONE handle. Through which ALL requests will be sent.
In UM there are ‘guardrails’ on completion currency. These actually hamper the performance of the highest performing applications but are there for the safety of many other applications. In KM you are presumed to understand multi-threading and concurrency and how to provide appropriate safeguards of your own