Device Object Access for Virtual Serial Driver

NTDEV
1

Hi, I am new to this forum, I found it by coincidence when looking for more information on windows drivers.
In advance, I am not an professional but am experienced development of software, I read in the FAQ that this is a good idea to mention early.

I am currently working on an virtual serial (COM port) driver for an project.
I read a lot about it, and what I do seems to be not the usual "virtual com driver" which just creates two connected virtual ports.

What I need is an driver which lets an user level application create an virtual port on demand, which is available for other applications to use, and all write and read requests including flow control signals should be passed to that application.

I know this could be also achieved using said common drivers, but I want to use this chance to learn more about windows drivers, so far I managed to write an virtual serial port driver where the writes just end in the void and read always returns an fixed string.

My question now is, how can make my application comunicate with the serial port without opening it using CreateFile, since this would block it from being accessed by external programs.
From what I found out about drivers in general, pretty much all communication between applications and drivers are happening trough IOCTL-Codes, which I already implemented for the normal serial port operation and I could just add some more for custom read and writes to the internal buffers, and to read the flow control signals, but for that the application needs to be able to get an handle for the device, without blocking other applications to do the same.

Coincidentally there happens to be also an another related problem, which actually prevents other aplications from being blocked when the port is already opened by one process.
For some reason I am able to open the port in as many applications as I want, which is not the behavior of normal COM ports.

I would appreciate if someone could help me figure this out, an link to some better documentation would also help, I could not find much to this, probably because this is not a common requirement for drivers.

2

If you have not called WdfDeviceInitSetExclusive, then your device can be opened by multiple applications. You could add a special ioctl that your controller calls to identify it as the "driver".

(The device can also be made "exclusive" by setting a registry value in the INF. Check https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/wdfdevice/nf-wdfdevice-wdfdeviceinitsetexclusive.)

3

Ah, I read about the registry entry in the INF, but I could find absolutely no information on what I need to write in the INF to make it happen.
An AI I asked about it said I need to add
[Device]
Exclusive = 1
SharedMode = 0

But this did not do anything.
WdfDeviceInitSetExclusive was mentioned nowhere, thanks for the info.
Weird is that the sample driver I looked at before did neither had an special INF entry (at least I did not see it) nor this function, and it behaved like an exclusive device, so I tought it should be the default behavior.

The question that now remains is, how can I let an application read and write to the internal buffers and registers without preventing an second application for "claiming" the port ?
The ideal solution would be to allow exact two handles for the port.
Since one application is pretty much guaranteed to run, to the user it would behave like an normal port from then on.

I remember somewhere that you can implement the "exclusiveness" in code as well, maybe it would be possible to do that and just add a counter that blocks further requests if two handles are active ?

4

I just tried to implement the function in my driver, but the documentation said its only for KMDF, and I indeed cant find it in my IDE.

I forgot to mention that at the beginning, but I started the driver as UMDF project, since the docs suggested you should only use KMDF if needed.

Is this enough of a reason to switch to KMDF, or is there an other way of getting the exclusive behavior ?
Preferably with the mentioned 2 applications limit.

5

If you are basing your driver on the microsoft virtual serial port sample, then the com ports created are not exclusive, unless you open them with no shared access.

If you are not using the sample, you should. It works.

6

I open both my ports and the one from the sample using GENERIC_READ | GENERIC_WRITE with no shared flags.
The sample driver could only be opened by one application this way.
My ports can be opened in an unlimited number of applications, using the same CreateFileA call.

I need a way to limit in on my ports too.
An example for the INF AddRegistry statement would help me, I can not find any information about that.

7

Just wanted to chime in and suggest if you do go with a user mode driver, Microsoft's current guidance is to use UMDF2 rather than UMDF.

Also, do not trust AIs for answers about UMDF2 drivers; Microsoft's Copilot, at least, confidently doles out wrong information. I don't think it has enough training data to be useful, yet.

8

I am already using UMDF2, I also already know that AI answers are not reliable, but it was the only way I could get at least some hints, although these seem to be wrong too.

9

OK, so I think I understand that you based your driver on the sample, and the sample works the way you want, which is that each com port is an exclusive device, and your driver behaves incorrectly, allowing multiple concurrent open handles.

So you have done something different from the sample. I'd start looking there.

10

And just to further muddy the waters. I tested my virtual serial port, and I can have a port opened by an application (windbg in my case) and I can use my control program to send ioctls to the port at the same time, to query runtime stats for example.

11

I'd suggest to create more than one device and have a dedicated symbolic link to the second device used to configure the primary (COM) device.

12

I did look into what I made different than the sample, but I could not find anything in the Code.
The INF file is different, but without any documentation of what all the stuff in it means, there is no way for me to determine what I need to change.
Are there any good tutorials or documentation on driver INF files, specifically mentioned AddRegistry-Directives ?

The windows docs just says "they are there", nothing more, nothing on how they look and how to use them.

I found some tutorials on how to write an basic INF file, which pretty much is what I got when I started the project with the UMDF2 template.

13

I thought about this first too, but it seemed simpler to have only one device, since I could basically add a separate set of IOCTL commands to control the port from the application while letting the others being used by the actual serial port implementation.

I could also easily add more ports and remove ones using pnputil or devdon, without any additional code.
I could not find any info on how you would go about making two devices which communicate with each other.

Sadly its not like there are many open source drivers to look on, they usuell are closed source since they usually come from larger companies.
And virtual device drivers seem to be especially rare.

14

I hope it is obvious that these lines do exactly the opposite of what you want.

15

Huh, why ?
Setting “Exclusive” to TRUE (1) and “ShareMode” to FALSE (0) seems like it should prevent multiple processes from opening the port at the same time.

Maybe it was confusing that I mentioned that I want exactly two processes to open the port later on.
This is something I wanted to implement later if it is possible.

16

Right. The only options are “1 only” or “unlimited”. If you want 2, then you cannot be exclusive. You would have to enforce the limits in your CreateFile handler. That can certainly be done, by passing extra info in the file name.

17

I think I got my driver mostly functional using the separate IOCTL’s approach.
The fact that it allows an arbitrary number of processes to open the port at the same time is actually not that big of a deal, it might even be useful for certain scenarios.
I would still like to know how the number of handles can be restricted in code.

Like, what exactly is the CreateFile handler ?
So far I only saw the EvtDeviceAdd handler as main “entry point” for an new device.
Also, what kind of additional info would I need to pass, and how ?
Are there any examples I could look at ?

18

Because the file object callbacks are not used as often, there is a separate API to register them. Check WdfDeviceInitSetFileObjectConfig. That gets you CREATE, CLEANUP and CLOSE.

19

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.