Win10 x64 UEFI boot (all updates) crashes on boot due to Verifier turned on
on a volume UpperFilter (inbox VolSnap.sys)
no non-Microsoft drivers in the kernel
the crash occurs due to IoMarkIrpPending/->PendingReturned misbehavior in inbox iorate.sys
(which is a volume LowerFilter)
the crash ONLY occurs if the Verifier setting of “Force pending I/O requests” is ON
WINDOWS VERSION DETAILS:
1: kd> vertarget
Windows 10 Kernel Version 14393 MP (4 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 14393.447.amd64fre.rs1_release_inmarket.161102-0100
Machine Name:
Kernel base = 0xfffff80109a84000 PsLoadedModuleList = 0xfffff80109d89060
Debug session time: Thu Dec 8 19:35:56.349 2016 (UTC + 3:00)
System Uptime: 0 days 0:00:01.301
REPRO:
no need in any non-MS drivers
set up Verifier as:
all settings to ON except
Randomized low resources simulation
Systematic low resources simulation
Code integrity checks
verify the inbox VolSnap.sys (volume UpperFilter) only
ADDITIONAL INFO:
turning Verifier ON on the 3rd party volume UpperFilter (which is above VolSnap)
causes the same crash
turning Verifier ON on iorate.sys (LowerFilter) or volume.sys (FDO) does NOT cause a crash
turning OFF the Verifier setting of “Force pending I/O requests”
does NOT crash
I enabled verifier on intelppm.sys and amdppm.sys some years back, without my filter driver running on them, and they both crashed. This was on windows 7 AFAIR. So its nothing new Peter.
As we discussed last week, the windows 10 verifier has just redefined the way status is returned from dispatch completion routines, and I remember having to rework the handling of IoMarkIrpPending() when I put an old XP driver through windows 7 verifier. Perhaps this is an instance of the same thing?
Sigh… With all due respect, neither of the drivers you mentioned were written by MSFT (as far as I know).
I don’t understand how a driver can get checked-in that crashes the system when Verifier is enabled. I *get* that the bug is in driver Y, but driver X crashes when DV is enabled. What I don’t get his how the dev owner of X can not have seen this. Unless there were changes to Y (that cause the problem) AFTER the last time X was changed.
Well, no. Verifier didn’t “redefine” anything… it now CHECKS for something that it never checked for before. That’s just a good thing. If you had to re-work how you handled IoMarkIrpPending in Win7, that’s because you had a bug in your code… no because Driver Verifier changed how the OS works.
> If somebody other than Mr. Shatskih can independently verify this, it’s pretty
big news. Positively scary.
Just verified and found it (BSOD with verifier enabled on volsnap.sys) to be true on a fresh install on Windows 10 Enterprise 10.0.14393
I first tried with /volatile and only “Force pending I/O requests” on volsnap.sys followed by some IOs (as the claim was that BSOD is for iorate.sys which was listed under LowerFilters of Volume Class GUID - HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class{71a27cdd-812a-11d0-bec7-08002be2092f})
That did not cause the crash. After that, enabled non-volatile with Standard settings + Force pending and rebooted the machine to find that the BSOD does happen.
Sigh… With all due respect, neither of the drivers you mentioned were written
by MSFT (as far as I know).
Since the thread title said “inbox-only drivers” and not “MSFT only drivers”, so Mr. Matt’s point may still be valid. The 2 drivers (intelppm.sys and amdppm.sys) carry MSFT copyright though.
@Peter
No need to sigh, I was just pointing out that even big companies don’t do things correctly. I have seen this so often, Sony, NVidia… How do devs not notice? Either they didn’t use verifier when they developed the driver or (and it was actually on windows 8 these CPU ppm drivers crashed, not win 7 as I originally stated) that Verifier tightened the rules, as we just discussed last week with that windows 10 crash on IoStatus handling.
Re IoMarkIrpPending, yes, it was wrong, the original dev hadn’t read about those routines that return status more processing, but it is another example of how Verifier continuously tightens up the rules. Strangely though the bug didn’t cause any apparent issues, and the driver was widely used.
I am not bitching about verifier though, I love it, its a great tool of course.
Quite so. It’s not about the company, its about the individual dev.
The root of the issue here is that writing kernel mode drivers is complex, and subtle, no matter how you look at it. There’s a vast gap between having a driver that “works” under all the scenarios you test, and having s driver that’s “correct” and architecturally sound. This gap is *much*smaller in WDF than it is in WDM, but it’s still there. The things that help close this gap are DV, SDV, and some of the more clever CA chicks. That’s why these are such valuable tools.
I’m a firm believer in the fact that even an experienced dev can benefit significantly from taking a good driver development class that emphasizes the architectural aspects of the I/O Subsystem. I’m sure I’m more than a little biased in this regard, though
I agree Peter, the architecture is hard to understand from the DDK documentation. Oney’s book is far better (but even his drivers have bugs, having just worked on one of his. )
Over here in Europe, we are cut off from the kind of information your company provides, which makes it a very slow learning process.
There has been quite a bit of “tension” over the years about the approach the WDK Docs should take. In The Beginning, the docs were designed to be an architectural tutorial and reference, with specific application of the DDIs. By NT V4, this approach was pretty much abandoned for new content due to the inherent difficulty and the fact that people, basically, just wanted to find out what function to call.
Hence the ancient book I wrote back in 1993, specifically to fill this gap.
We’ve been threatening to write another such book, but oriented to WDF, for many years. Sigh…
Folks should bear in mind that a flight from London to Boston is only 400 Euros… and we teach WDF almost every month OK… I’m going to stop breaking, or coming lose to breaking, the rules right now. No more self-promotion. I apologize to anyone offended… I’m really only making conversation here, not doing serious marketing.
I can understand your reluctance on WDF, but that is exactly the reason to
go to a good class. I’ve encountered a number of clients who try to push
me to WDM, after I have done the driver with WDF they don’t want to go back.
You can read all you want, but it is really understanding the mindset of
writing a WDF driver that makes it better. This is what a good class or a
conference provides.
One of the things feel Microsoft is blowing badly is not having a “real
WinHEC” or a Driver Developers Conference. I’ve been writing device drivers
commercially on and off for 42 years, every one of the WinHEC’s or DDC’s I
attended I learned something, either from the presentations, or just as
often as hanging out with the people on this forum.
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@hotmail.com
Sent: Monday, December 12, 2016 10:39 AM
To: Windows System Software Devs Interest List Subject: RE:[ntdev] Win10: crashes the Verifier with inbox-only drivers
"Folks should bear in mind that a flight from London to Boston is only 400 Euros… " And the hotel, meals etc, for two devs, for a week?
Try telling a manager that’s cheap.
And WDF? Yeah, I know its better, but I have so much invested in WDM, both in years and code library that I am not going to stop using it now…
The kind of windows kernel work we get in Europe isn’t often the kind of straightforward driver work that is what WDF is aimed at, instead it is often weird stuff, such as allowing a driver to be unloaded even though the app has a handle open on it, or the CPU speed management driver that had to keep the CPU throttled down even though an app was running full time (intercepts ACPI Irps, gets the range of speeds and associated values to write to the CPU port, sure you remember it, it caused quite a stir when I discussed it here.). Or now, a USB bridge (as well as an RT KS audio bridge) that ‘robs’ a device from appearing to user mode on one system, and sends the data down a USB connection to another.
Does WDF work for these kinds of *hack* drivers?
(And from what I did read of WDF it seemed to take care of power and pnp, the two really tricky aspects of drivers, but now I have a large library of boilerplate all my code in this respect is old and working well, so are there other big advantages to WDF?)
> (And from what I did read of WDF it seemed to take care of power and pnp, the two really tricky
aspects of drivers, but now I have a large library of boilerplate all my code in this respect is old and
working well, so are there other big advantages to WDF?)
Bus drivers.
IO target (handles the device being removed while your kmode code holds the open file on it).
And, aside of this - not much. The WdfRequestFormatXxx functions are in fact more arcane, lesser intuitive, and more limited then usual IRP/IrpSp filling.
Yes WDF will support strange drivers, over the years I’ve done a number of
drivers that were considered on the edge. In some ways you can think of WDF
as analogous to object oriented programming, it can help a lot though you
may want it in moderation.
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@hotmail.com
Sent: Monday, December 12, 2016 11:23 AM
To: Windows System Software Devs Interest List Subject: RE:[ntdev] Win10: crashes the Verifier with inbox-only drivers
@Don
The kind of windows kernel work we get in Europe isn’t often the kind of straightforward driver work that is what WDF is aimed at, instead it is often weird stuff, such as allowing a driver to be unloaded even though the app has a handle open on it, or the CPU speed management driver that had to keep the CPU throttled down even though an app was running full time (intercepts ACPI Irps, gets the range of speeds and associated values to write to the CPU port, sure you remember it, it caused quite a stir when I discussed it here.). Or now, a USB bridge (as well as an RT KS audio bridge) that ‘robs’ a device from appearing to user mode on one system, and sends the data down a USB connection to another.
Does WDF work for these kinds of hack drivers?
(And from what I did read of WDF it seemed to take care of power and pnp, the two really tricky aspects of drivers, but now I have a large library of boilerplate all my code in this respect is old and working well, so are there other big advantages to WDF?)
“Don Burn” wrote in message news:xxxxx@ntdev… > Yes WDF will support strange drivers, over the years I’ve done a number of > drivers that were considered on the edge. In some ways you can think of WDF > as analogous to object oriented programming, it can help a lot though you > may want it in moderation. > > > Don Burn > Windows Driver Consulting > Website: http://www.windrvr.com > > > > > -----Original Message----- > From: xxxxx@lists.osr.com > [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@hotmail.com > Sent: Monday, December 12, 2016 11:23 AM > To: Windows System Software Devs Interest List > Subject: RE:[ntdev] Win10: crashes the Verifier with inbox-only drivers > > @Don > > The kind of windows kernel work we get in Europe isn’t often the kind of > straightforward driver work that is what WDF is aimed at, instead it is > often weird stuff, such as allowing a driver to be unloaded even though the > app has a handle open on it, or the CPU speed management driver that had to > keep the CPU throttled down even though an app was running full time > (intercepts ACPI Irps, gets the range of speeds and associated values to > write to the CPU port, sure you remember it, it caused quite a stir when I > discussed it here.). Or now, a USB bridge (as well as an RT KS audio > bridge) that ‘robs’ a device from appearing to user mode on one system, and > sends the data down a USB connection to another. > > Does WDF work for these kinds of hack drivers? > > (And from what I did read of WDF it seemed to take care of power and pnp, > the two really tricky aspects of drivers, but now I have a large library of > boilerplate all my code in this respect is old and working well, so are > there other big advantages to WDF?) > > > > — > NTDEV is sponsored by OSR > > Visit the list online at: > http: > > MONTHLY seminars on crash dump analysis, WDF, Windows internals and software > drivers! > Details at http: > > To unsubscribe, visit the List Server section of OSR Online at > http: > ></http:></http:></http:>
TL;DR? It’s worth it to move from WDM to WDF… even if you’re a very experienced WDM dev with a ton of proven code. I did it back in 2005 or whenever… and I don’t want to ever go back.
The Long Version:
Back in 2005, I was in precisely the same place that you are now. I though I had a pretty good handle on WDM, and certainly had a reasonable amount of code under my belt that I could cut/paste as required to do what I needed.
There are several things that drove me to WDF, however:
It’s brief: You can write a fully working driver for a piece of real hardware in 1200 or so lines of code. I challenge you to write the WDM code to handle JUST your PnP and Power requirements in that number of lines. IIRC, 1200 lines of code for PnP/Power was the MINIMUM I could get away with in WDM… and that was basically code that did nothing other than Keep Windows Happy.
It’s mostly well thought-out and pleasant to use: It either handles for you or provides a tidy model for handling a lot of the annoying shite that one must constantly deal with in WDM. Queuing IRPs is a classic example. Ugh. I found that WDF lets me get on with worrying about my device faster, spending less time worry about Keeping Windows Happy.
There’s less legacy stuff to drag along: With each release of Windows, I feel like in WDM there’s some much old, legacy, stuff that I’m encumbered by. Good decision, bad decisions, really no matter. They’re like the chain worn by Jacob Marley’s Ghost in A Christmas Carol (http://paranoiastrikesdeep.blogspot.com/2013/12/i-wear-chain-i-forged-in-life.html). Remember to set FILE_DEVICE_SECURE_OPEN? Remember under which circumstances the “Exclusive” argument to IoCreateDevice actually does what it says it does? You can’t avoid ALL the silliness, but you can avoid a lot of it with WDF.
The debugging/checking/development process is far superior by design: You really notice this when you teach a group of devs to write Windows drivers. In WDM class, the development, test, fix, re-test process usually includes a regular progression of Blue Screens. In WDF class, this same process most often avoids Blue Screens, and instead you get back some sort of error from an API that you call. Not all the time, but MUCH of the time. And when you DO get a BSOD, it’s much easier to figure out what went wrong.
I could go on all day. Hmmmm… sometimes I do. But, in any case, as a WDM developer with a reasonable amount of experience, I switched to WDF a little more than 10 years ago… and it is my goal and practice to never write a WDM driver where a WDF driver will do.
