Help Stamp Out Sensless WDM Usage

On Thu, Mar 8, 2018 at 2:46 AM, xxxxx@hotmail.com
wrote:
>> The names are different, the patterns not so much.
>
>
> …which means that your knowledge of both Linux and Windows kernel-mode programming is critically close to zero.In fact, it is hard too imagine two systems that are THAT drastically different from one another, both architecturally and philosophically…
>
>

Can you provide a concrete example? I’ve had what might be a similar
argument presented to me about IO completion ports, or
WaitForMultipleObjects, but despite the specifics of the API
equivalent information is available on both systems. In the case of
IO, they tend to view things from the opposite side.

>> E.g. does Microsoft have some kind of monopoly on asynchronous IO?
>
> Actually, you are almost there.Indeed, Windows NT was the first OS to introduce an asynch IO model that was absolutely generic for all device types.UNIX-like systems have not had had something like that for AT LEAST one more decade. Historically they had been making a distinction between asynch pipe/socket IO with select()/poll() semantics, and asynch disk IO that is concerned about the actual IO completion, rather than data/buffer space availability the way pipes and sockets do. Although the former was the same across all UNIX-like systems, the latter was vendor-specific.
>
> Solaris was the first UNIX-like system to introduce Windows-like IO completion ports, and FreeBSD followed the same path with introducing kqueue that relies upon the same principles.
> Linux still does not really have it - its epoll() does not offer the same functionality that kqueue and Solaris completion ports do.
>

Importantly epoll can support these facilities. For some of them it
does, such as timerfds. In other cases the functionality is elsewhere
for historical reasons.

>
> BTW, once we are at it, check the Linux kernel and see how many devices other than pipes and sockets implement asynch IO handlers. Out of those who do, check how many of them actually make use of kernel asynch IO model (i.e. make use of iocb) , rather than just delegating the whole thing to a dedicated thread that, in actuality, works on synchronous basis (i.e. implement it the same way that GNU C library does in the userland)
>

So are you able to comment on the Windows implementation as a comparison?

Don’t know if you meant “disappearing” instead of “disparaging”… but not nearly enough. I’d like MORE disappearing AND active disparaging of WDM. There’s still WAAAAY too much legacy documentation, and as our colleague Mr. R0b0t1 stated, people file “curated” WDM samples (that mostly suck, but whatever) and then have enough MSDN to move those samples further into the world of fucked-up.

When would someone HAVE to write a WDM driver, exactly?

Good. AFAIC, they should need a secret decoder ring and a personal permission slip signed in Joe Belfiore’s blood before they can write a WDM driver.

OK, maybe I’m exaggerating a LITTLE…

But shitty filters written in WDM by are now an actual, quantifiable, problem in terms of system stability (and upgrade-ability). I’m soooo fucking tired of seeing newbs posting WDM questions and getting a specific, pointed, answer, and then going away to wreck unintentional havoc. It’s not that they WANT to do the wrong thing. They just have no way of knowing that’s what they’re doing. No reasonable dev WOULD know. That’s the fatal flaw of WDM in the 21st Century, and why people need to learn to “just say no” to WDM. Hence my (continued) rant.

Peter
OSR
@OSRDrivers

I can – And I’ll even stop ranting for a moment while I do it.

In Windows, it’s generally considered to be “bad form” to synchronously wait for a device to do your bidding. The driver model is designed to be basically asynchronous and event-driven:

  1. You get a request (read, write, IOCTL).

2a) If the device is free, you program the device to perform the requested I/O operation and save some context for the pending request.

2b) If the device is not free, you queue the request and RETURN from your driver.

  1. Assuming you start the request on the device, your driver code then RETURNS, leaving the request in progress on the device.

  2. The device interrupts to indicate the request has been completed. Your driver queues a DpcForIsr from this ISR.

  3. In your DpcForIsr you retrieve the context for the pending request that you saved in step 2a, above. You complete whatever processing is required for the request (if you have a DMA type device, there’s probably very little to do… if you have a PIO type device and this is a READ operation from the user, you probably have to move the data from the device to the user’s data buffer). Then you complete the request.

  4. With the request completed, and your device now “idle”, you look to see if you have any pending requests queued for your device that you can now start.

I think the model is simplicity itself :wink:

Peter
OSR
@OSRDrivers

The only time in recent memory I have had to write a WDM driver was to
develop a filter for a bus driver. But these are so rare that it should not
matter.

Don Burn
Windows Driver Consulting
Website: http://www.windrvr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@osr.com
Sent: Friday, March 09, 2018 10:21 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Help Stamp Out Sensless WDM Usage



Don’t know if you meant “disappearing” instead of “disparaging”… but not
nearly enough. I’d like MORE disappearing AND active disparaging of WDM.
There’s still WAAAAY too much legacy documentation, and as our colleague Mr.
R0b0t1 stated, people file “curated” WDM samples (that mostly suck, but
whatever) and then have enough MSDN to move those samples further into the
world of fucked-up.



When would someone HAVE to write a WDM driver, exactly?



Good. AFAIC, they should need a secret decoder ring and a personal
permission slip signed in Joe Belfiore’s blood before they can write a WDM
driver.

OK, maybe I’m exaggerating a LITTLE…

But shitty filters written in WDM by are now an actual, quantifiable,
problem in terms of system stability (and upgrade-ability). I’m soooo
fucking tired of seeing newbs posting WDM questions and getting a specific,
pointed, answer, and then going away to wreck unintentional havoc. It’s not
that they WANT to do the wrong thing. They just have no way of knowing
that’s what they’re doing. No reasonable dev WOULD know. That’s the fatal
flaw of WDM in the 21st Century, and why people need to learn to “just say
no” to WDM. Hence my (continued) rant.

Peter
OSR
@OSRDrivers


NTDEV is sponsored by OSR

Visit the list online at:
http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software
drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at
http:</http:></http:></http:>

Good one, thank you for that. We did that here recently as well.

Rare and unsupported as well, we should note.

Peter
OSR
@OSRDrivers

shitty devs " wreck[ing] unintentional havoc" would be a good thing. It is
when they wreak havoc that things go wrong.

Mark Roddy

On Fri, Mar 9, 2018 at 10:50 AM, xxxxx@osr.com
wrote:

>


>
> Good one, thank you for that. We did that here recently as well.
>
> Rare and unsupported as well, we should note.
>
> Peter
> OSR
> @OSRDrivers
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:> showlists.cfm?list=ntdev>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:></http:>

Ducking auto-correct, I swear. I’ll go back and edit that post. Oh, wait…

Peter
OSR
@OSRDrivers

registry filter drivers. I suppose you could write one using KMDF but it
would add unneeded complexity rather than reducing it. Also you could write
a bus filter driver using KMDF and then escape into WDM to do all the bus
filtering :slight_smile:

Mark Roddy

On Fri, Mar 9, 2018 at 10:50 AM, xxxxx@osr.com
wrote:

>


>
> Good one, thank you for that. We did that here recently as well.
>
> Rare and unsupported as well, we should note.
>
> Peter
> OSR
> @OSRDrivers
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:> showlists.cfm?list=ntdev>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:></http:>

Registry Filters, Object Manager Filters, Process Manager Filters… these
are all Kernel Service, which I already mentioned are an entirely separate
category of beast. WDM for this use is entirely fine.

Peter
OSR
@OSRDrivers

On Sat, Mar 10, 2018 at 10:43 AM, xxxxx@gmail.com <
xxxxx@lists.osr.com> wrote:

registry filter drivers. I suppose you could write one using KMDF but it
would add unneeded complexity rather than reducing it. Also you could write
a bus filter driver using KMDF and then escape into WDM to do all the bus
filtering :slight_smile:

Mark Roddy

On Fri, Mar 9, 2018 at 10:50 AM, xxxxx@osr.com
> wrote:
>
>>


>>
>> Good one, thank you for that. We did that here recently as well.
>>
>> Rare and unsupported as well, we should note.
>>
>> Peter
>> OSR
>> @OSRDrivers
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> Visit the list online at: http:>> lists.cfm?list=ntdev>
>>
>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
>> software drivers!
>> Details at http:
>>
>> To unsubscribe, visit the List Server section of OSR Online at <
>> http://www.osronline.com/page.cfm?name=ListServer&gt;
>>
>
> — NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars
> on crash dump analysis, WDF, Windows internals and software drivers!
> Details at To unsubscribe, visit the List Server section of OSR Online at</http:></http:>

I think a HUGE problem for a newbie Window’s driver writer’s is figuring out he correct kind of driver to write. Other platforms tend not to have all the different flavors, they just have “a driver”. Even an experienced developer like myself sometimes has to look carefully and do a lot of digging what kind of driver is needed. For example, a network kernel winsock client that does not process IRPs would be more of a kernel service, but one that can be opened by an application (meaning it can process IRPS) would more likely be something like a root-enumerated PnP driver. If I had a kernel service that I want to control from powershell via WMI method calls, I would need to make it a driver with a do almost nothing virtual device object, just so the OS has someplace to send WMI IRPS.

Perhaps something like a decision tree in the WDK documentation on how to pick the correct kind of driver would help. A newbie will look at this thread and be clueless how a “driver”, written to the WDM API (to be avoided), is a different thing than a “kernel service” written to the WDM API (which is acceptable). This terminology is not spelled out, it’s just a set of slightly fuzzy categories that have informally formed over the years.

Another thing that could help newbie driver developers would be if the VisualStudio driver wizard knew how to make a variety of specialized class drivers, based on the latest thinking on what’s “optimal”. So then there would be no need to hunt through the samples to find something sort of similar. You would just say dear VS Wizard, make me generic storage controller driver, and it would (today) spit out a storport driver for a ramdisk or something, and tomorrow it might spit our a KMDF driver with the extra storage support.

Even though I see lots of value in miniports, and other class specific frameworks, I wonder if these just create confusion and ignorance in the long term. Windows kernel developers tend to have boundaries on their driver interface knowledge, significantly due to the miniport model. If you work for a large corporation on the same project for many years, the miniport model tends to keep you ignorant about anything outside your little world. There might be some shift occurring in this, as it seems like Microsoft is moving toward a NIC miniport being a KMDF driver, with some network specific queue ring interfaces.

Jan

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@gmail.com
Sent: Saturday, March 10, 2018 9:52 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Help Stamp Out Sensless WDM Usage

Registry Filters, Object Manager Filters, Process Manager Filters… these are all Kernel Service, which I already mentioned are an entirely separate category of beast. WDM for this use is entirely fine.

Peter
OSR
@OSRDrivers

On Sat, Mar 10, 2018 at 10:43 AM, xxxxx@gmail.commailto:xxxxx > wrote:
registry filter drivers. I suppose you could write one using KMDF but it would add unneeded complexity rather than reducing it. Also you could write a bus filter driver using KMDF and then escape into WDM to do all the bus filtering :slight_smile:

Mark Roddy

On Fri, Mar 9, 2018 at 10:50 AM, xxxxx@osr.commailto:xxxxx > wrote:



Good one, thank you for that. We did that here recently as well.

Rare and unsupported as well, we should note.

Peter
OSR
@OSRDrivers


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:

— NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers! Details at To unsubscribe, visit the List Server section of OSR Online at

— NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers! Details at To unsubscribe, visit the List Server section of OSR Online at</http:></http:></http:></mailto:xxxxx></mailto:xxxxx>

On Sat, Mar 10, 2018 at 6:03 PM, xxxxx@pmatrix.com > wrote:

>
>
>
> Perhaps something like a decision tree in the WDK documentation on how to
> pick the correct kind of driver would help. A newbie will look at this
> thread and be clueless how a “driver”, written to the WDM API (to be
> avoided), is a different thing than a “kernel service” written to the WDM
> API (which is acceptable). This terminology is not spelled out, it’s just a
> set of slightly fuzzy categories that have informally formed over the years.
>
>
>
>
>
Um, you mean like this: Choosing a driver model
https:

Mark Roddy</https:>

Peter wrote: “When would someone HAVE to write a WDM driver, exactly?”

Kernel Service drivers. I have had to write a few in my time, in fact wrote one last year.

I left software development about 12 years ago (before WDF was a thing). Since I have returned, I’ve written one new driver that was a virtual storport miniport. These beasts live somewhere between a WDM and a miniport.

Recently I have been working on a older WDM driver written by someone else about 10 years ago. I have been considering the idea of porting the driver to a WDF/KMDF driver, but have put that on the back burner because of the demands to get code shipped.

However, after reading Peter’s post, I have decided to move the port to the foreground. How difficult can it be; right?

After reading the documentation, I see that if you want to receive PNP notifications, you must be a WDM driver. In the driver I am working on, it filters IRP_MJ_PNP in order to detect special files on the volume (this is a volume filter driver, not an FS volume filter). This is done to maintain the power pageable bit while the filter is active.

Does this requirement prevent me from porting to WDF/KMDF?

– Jamey

BTW: When I reply via email, the emails get lost. I sent this same post via email, and the email is nowhere to be found, not even in my outbox. Strange. So I have to come here to the website to post. If this gets posted twice, I apologize.

>>When would someone HAVE to write a WDM driver, exactly

Kernel Service drivers.

Yes, absolutely agree that writing Kernel Services using WDM is both entirely appropriate and a Best Practice. I *thought* I said that in my long initial rant, but perhaps it was somewhere else that I ranted this carve-out.

But, in any case, agreed: Kernel Services should be WDM (though they do not HAVE to be WDM).

Peter
OSR
@OSRDrivers

Peter,

You did say it, and for a simple kernel service I totally agree. I
recently did rewrite one kernel service driver to KMDF, the original WDM had
something like 20 cancel safe queues to do weird things. Moving it to KMDF
queues made the code so much nicer!

Don Burn
Windows Driver Consulting
Website: http://www.windrvr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@osr.com
Sent: Monday, March 12, 2018 11:55 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Help Stamp Out Sensless WDM Usage

>>When would someone HAVE to write a WDM driver, exactly

>Kernel Service drivers.

Yes, absolutely agree that writing Kernel Services using WDM is both
entirely appropriate and a Best Practice. I thought I said that in my
long initial rant, but perhaps it was somewhere else that I ranted this
carve-out.

But, in any case, agreed: Kernel Services should be WDM (though they do not
HAVE to be WDM).

Peter
OSR
@OSRDrivers


NTDEV is sponsored by OSR

Visit the list online at:
http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software
drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at
http:</http:></http:></http:>

Warning: Thread Drift Ahead

No! You can still filter and receive these. WdfDeviceInitAssignWdmIrpPreprocessCallback is your friend.

You are aware that there’s WdfDeviceSetSpecialFileSupport ?

Peter
OSR
@OSRDrivers

xxxxx@osr.com wrote:

No! You can still filter and receive these. WdfDeviceInitAssignWdmIrpPreprocessCallback is your friend.

I suspect the author of the original quote really meant “at least a WDM
driver”.  All KMDF drivers are also WDM drivers.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Peter,

Does KMDF support normal flag handling for filter drivers? I thought
it did, so since the original question was on a filter, this should be let
the framework do it.

Don Burn
Windows Driver Consulting
Website: http://www.windrvr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@osr.com
Sent: Monday, March 12, 2018 12:01 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Help Stamp Out Sensless WDM Usage

Warning: Thread Drift Ahead



No! You can still filter and receive these.
WdfDeviceInitAssignWdmIrpPreprocessCallback is your friend.



You are aware that there’s WdfDeviceSetSpecialFileSupport ?

Peter
OSR
@OSRDrivers


NTDEV is sponsored by OSR

Visit the list online at:
http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software
drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at
http:</http:></http:></http:>

Peter, no. I just started digging into the documentation last night. I have zero WDF/KMDF experience. It will take me a few days to come up to speed.

I’ve run into my first issue :slight_smile:

In the WDM driver, I would check device characteristics to see if the device was something we were not interested in filtering. Here is a snippet of WDM code:

NTSTATUS FilterAddDevice(In PDRIVER_OBJECT Driver,
In PDEVICE_OBJECT PhysicalDevice) {
if (FlagOn(PhysicalDevice->Characteristics, FILE_REMOVABLE_MEDIA) ||
FlagOn(PhysicalDevice->Characteristics, FILE_READ_ONLY_DEVICE) ||
FlagOn(PhysicalDevice->Characteristics, FILE_READ_ONLY_VOLUME)) {
// We are not interested in removable media. Exit with nothing to do.
return STATUS_SUCCESS;
}

This would skip filtering read-only and removable devices. In WDF, I cannot get access to the lower device object to get the characteristics. I assume WdfDeviceCreate() attaches to the lower device by me having called WdfFdoInitSetFilter() prior to calling WdfDeviceCreate().

So, let’s say I have called WdfDeviceCreate(), I check the characteristics after calling WdfGetDeviceCharacteristsic(), and determine I really do not want to filter this device.

How do I go about detaching from the lower device before exiting EvtDevicedd() function? I don’t want to have a null filter sitting there just passing data along; do I?

Maybe I am missing something. I am reading the documentation while trying to do the port.

– Jamey