Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Before Posting...
Please check out the Community Guidelines in the Announcements and Administration Category.

Win10: crashes the Verifier with inbox-only drivers

Maxim_S._ShatskihMaxim_S._Shatskih Member Posts: 10,396
Win10 x64 UEFI boot (all updates) crashes on boot due to Verifier turned on
on a volume UpperFilter (inbox VolSnap.sys)
no non-Microsoft drivers in the kernel
the crash occurs due to IoMarkIrpPending/->PendingReturned misbehavior in inbox iorate.sys
(which is a volume LowerFilter)
the crash ONLY occurs if the Verifier setting of "Force pending I/O requests" is ON

WINDOWS VERSION DETAILS:
1: kd> vertarget
Windows 10 Kernel Version 14393 MP (4 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 14393.447.amd64fre.rs1_release_inmarket.161102-0100
Machine Name:
Kernel base = 0xfffff801`09a84000 PsLoadedModuleList = 0xfffff801`09d89060
Debug session time: Thu Dec 8 19:35:56.349 2016 (UTC + 3:00)
System Uptime: 0 days 0:00:01.301

REPRO:
no need in any non-MS drivers
set up Verifier as:
all settings to ON except
Randomized low resources simulation
Systematic low resources simulation
Code integrity checks
verify the inbox VolSnap.sys (volume UpperFilter) only

ADDITIONAL INFO:
turning Verifier ON on the 3rd party volume UpperFilter (which is above VolSnap)
causes the same crash
turning Verifier ON on iorate.sys (LowerFilter) or volume.sys (FDO) does NOT cause a crash
turning OFF the Verifier setting of "Force pending I/O requests"
does NOT crash

Comments

  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 7,284
    Wow.

    If somebody other than Mr. Shatskih can independently verify this, it's pretty big news. Positively scary.

    Peter
    OSR
    @OSRDrivers

    Peter Viscarola
    OSR
    @OSRDrivers

  • matt_sykesmatt_sykes Member - All Emails Posts: 270
    I enabled verifier on intelppm.sys and amdppm.sys some years back, without my filter driver running on them, and they both crashed. This was on windows 7 AFAIR. So its nothing new Peter.

    As we discussed last week, the windows 10 verifier has just redefined the way status is returned from dispatch completion routines, and I remember having to rework the handling of IoMarkIrpPending() when I put an old XP driver through windows 7 verifier. Perhaps this is an instance of the same thing?
  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 7,284
    <quote>
    So its nothing new Peter
    </quote>

    Sigh... With all due respect, neither of the drivers you mentioned were written by MSFT (as far as I know).

    I don't understand how a driver can get checked-in that crashes the system when Verifier is enabled. I *get* that the bug is in driver Y, but driver X crashes when DV is enabled. What I don't get his how the dev owner of X can not have seen this. Unless there were changes to Y (that cause the problem) AFTER the last time X was changed.

    <quote>
    the windows 10 verifier has just redefined the way
    status is returned from dispatch completion routines
    </quote>

    Well, no. Verifier didn't "redefine" anything... it now CHECKS for something that it never checked for before. That's just a good thing. If you had to re-work how you handled IoMarkIrpPending in Win7, that's because you had a bug in your code... no because Driver Verifier changed how the OS works.

    Peter
    OSR
    @OSRDrivers

    Peter Viscarola
    OSR
    @OSRDrivers

  • Anand_AAnand_A Member - All Emails Posts: 66
    > If somebody other than Mr. Shatskih can independently verify this, it's pretty
    > big news. Positively scary.

    Just verified and found it (BSOD with verifier enabled on volsnap.sys) to be true on a fresh install on Windows 10 Enterprise 10.0.14393
    I first tried with /volatile and only "Force pending I/O requests" on volsnap.sys followed by some IOs (as the claim was that BSOD is for iorate.sys which was listed under LowerFilters of Volume Class GUID - HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{71a27cdd-812a-11d0-bec7-08002be2092f})
    That did not cause the crash. After that, enabled non-volatile with Standard settings + Force pending and rebooted the machine to find that the BSOD does happen.

    > Sigh... With all due respect, neither of the drivers you mentioned were written
    > by MSFT (as far as I know).
    Since the thread title said "inbox-only drivers" and not "MSFT only drivers", so Mr. Matt's point may still be valid. The 2 drivers (intelppm.sys and amdppm.sys) carry MSFT copyright though.
  • matt_sykesmatt_sykes Member - All Emails Posts: 270
    @Peter
    No need to sigh, I was just pointing out that even big companies don't do things correctly. I have seen this so often, Sony, NVidia... How do devs not notice? Either they didn't use verifier when they developed the driver or (and it was actually on windows 8 these CPU ppm drivers crashed, not win 7 as I originally stated) that Verifier tightened the rules, as we just discussed last week with that windows 10 crash on IoStatus handling.

    Re IoMarkIrpPending, yes, it was wrong, the original dev hadn't read about those routines that return status more processing, but it is another example of how Verifier continuously tightens up the rules. Strangely though the bug didn't cause any apparent issues, and the driver was widely used.

    I am not bitching about verifier though, I love it, its a great tool of course.
  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 7,284
    <quote>
    I was just pointing out that even big companies don't do things
    correctly.
    </quote>

    Quite so. It's not about the company, its about the individual dev.

    The root of the issue here is that writing kernel mode drivers is complex, and subtle, no matter how you look at it. There's a vast gap between having a driver that "works" under all the scenarios you test, and having s driver that's "correct" and architecturally sound. This gap is *much*smaller in WDF than it is in WDM, but it's still there. The things that help close this gap are DV, SDV, and some of the more clever CA chicks. That's why these are such valuable tools.

    I'm a firm believer in the fact that even an experienced dev can benefit significantly from taking a good driver development class that emphasizes the architectural aspects of the I/O Subsystem. I'm sure I'm more than a little biased in this regard, though :-)

    Peter
    OSR
    @OSRDrivers

    Peter Viscarola
    OSR
    @OSRDrivers

  • matt_sykesmatt_sykes Member - All Emails Posts: 270
    I agree Peter, the architecture is hard to understand from the DDK documentation. Oney's book is far better (but even his drivers have bugs, having just worked on one of his. :) )

    Over here in Europe, we are cut off from the kind of information your company provides, which makes it a very slow learning process.
  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 7,284
    <quote>
    the architecture is hard to understand from the DDK
    documentation.
    </quote>

    There has been quite a bit of "tension" over the years about the approach the WDK Docs should take. In The Beginning, the docs were designed to be an architectural tutorial and reference, with specific application of the DDIs. By NT V4, this approach was pretty much abandoned for new content due to the inherent difficulty and the fact that people, basically, just wanted to find out what function to call.

    Hence the ancient book I wrote back in 1993, specifically to fill this gap.

    We've been threatening to write another such book, but oriented to WDF, for many years. Sigh...

    Folks should bear in mind that a flight from London to Boston is only 400 Euros... and we teach WDF almost every month :-) OK.. I'm going to stop breaking, or coming lose to breaking, the rules right now. No more self-promotion. I apologize to anyone offended... I'm really only making conversation here, not doing serious marketing.

    Peter
    OSR
    @OSRDrivers

    Peter Viscarola
    OSR
    @OSRDrivers

  • matt_sykesmatt_sykes Member - All Emails Posts: 270
    "Folks should bear in mind that a flight from London to Boston is only 400
    Euros... " And the hotel, meals etc, for two devs, for a week?

    Try telling a manager that's cheap. :)

    And WDF? Yeah, I know its better, but I have so much invested in WDM, both in years and code library that I am not going to stop using it now...
  • Don_BurnDon_Burn Member - All Emails Posts: 1,653
    I can understand your reluctance on WDF, but that is exactly the reason to
    go to a good class. I've encountered a number of clients who try to push
    me to WDM, after I have done the driver with WDF they don't want to go back.
    You can read all you want, but it is really understanding the mindset of
    writing a WDF driver that makes it better. This is what a good class or a
    conference provides.

    One of the things feel Microsoft is blowing badly is not having a "real
    WinHEC" or a Driver Developers Conference. I've been writing device drivers
    commercially on and off for 42 years, every one of the WinHEC's or DDC's I
    attended I learned something, either from the presentations, or just as
    often as hanging out with the people on this forum.


    Don Burn
    Windows Driver Consulting
    Website: http://www.windrvr.com



    -----Original Message-----
    From: xxxxx@lists.osr.com
    [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@hotmail.com
    Sent: Monday, December 12, 2016 10:39 AM
    To: Windows System Software Devs Interest List <xxxxx@lists.osr.com>
    Subject: RE:[ntdev] Win10: crashes the Verifier with inbox-only drivers

    "Folks should bear in mind that a flight from London to Boston is only 400
    Euros... " And the hotel, meals etc, for two devs, for a week?

    Try telling a manager that's cheap. :)

    And WDF? Yeah, I know its better, but I have so much invested in WDM, both
    in years and code library that I am not going to stop using it now...

    ---
    NTDEV is sponsored by OSR

    Visit the list online at:
    <http://www.osronline.com/showlists.cfm?list=ntdev>;

    MONTHLY seminars on crash dump analysis, WDF, Windows internals and software
    drivers!
    Details at <http://www.osr.com/seminars>;

    To unsubscribe, visit the List Server section of OSR Online at
    <http://www.osronline.com/page.cfm?name=ListServer>;
  • Maxim_S._ShatskihMaxim_S._Shatskih Member Posts: 10,396
    >enabled. I *get* that the bug is in driver Y, but driver X crashes when DV is enabled.

    No, the bug is in iorate.sys (lower), but it manifests itself only if DV is turned on for volsnap.sys (upper).

    It is not known whether volsnap.sys has any bug, probably no.

    --
    Maxim S. Shatskih
    former Microsoft MVP on File System And Storage
    xxxxx@storagecraft.com
    http://www.storagecraft.com
  • Maxim_S._ShatskihMaxim_S._Shatskih Member Posts: 10,396
    BTW - I've got a private email from the MS's dev who owns iorate.sys, and he said he fixed the bug and checked in the fix.

    He does not know when this fix will appear to the public, but it _definitely_ will with the next major update after RS1.

    --
    Maxim S. Shatskih
    former Microsoft MVP on File System And Storage
    xxxxx@storagecraft.com
    http://www.storagecraft.com
  • Maxim_S._ShatskihMaxim_S._Shatskih Member Posts: 10,396
    >WDF almost every month

    WDF is good.

    But looks like ALL their analogs of IoBuildPartialMdl require mapping of the master MDL to system PTEs.

    If I'm correct with this (probably I have overlooked something) - then this is rather scary.

    --
    Maxim S. Shatskih
    Microsoft MVP on File System And Storage
    xxxxx@storagecraft.com
    http://www.storagecraft.com
  • matt_sykesmatt_sykes Member - All Emails Posts: 270
    @Don

    The kind of windows kernel work we get in Europe isn't often the kind of straightforward driver work that is what WDF is aimed at, instead it is often weird stuff, such as allowing a driver to be unloaded even though the app has a handle open on it, or the CPU speed management driver that had to keep the CPU throttled down even though an app was running full time (intercepts ACPI Irps, gets the range of speeds and associated values to write to the CPU port, sure you remember it, it caused quite a stir when I discussed it here.). Or now, a USB bridge (as well as an RT KS audio bridge) that 'robs' a device from appearing to user mode on one system, and sends the data down a USB connection to another.

    Does WDF work for these kinds of *hack* drivers?

    (And from what I did read of WDF it seemed to take care of power and pnp, the two really tricky aspects of drivers, but now I have a large library of boilerplate all my code in this respect is old and working well, so are there other big advantages to WDF?)
  • Maxim_S._ShatskihMaxim_S._Shatskih Member Posts: 10,396
    > (And from what I did read of WDF it seemed to take care of power and pnp, the two really tricky
    >aspects of drivers, but now I have a large library of boilerplate all my code in this respect is old and
    >working well, so are there other big advantages to WDF?)

    Bus drivers.

    IO target (handles the device being removed while your kmode code holds the open file on it).

    And, aside of this - not much. The WdfRequestFormatXxx functions are in fact more arcane, lesser intuitive, and more limited then usual IRP/IrpSp filling.

    --
    Maxim S. Shatskih
    Microsoft MVP on File System And Storage
    xxxxx@storagecraft.com
    http://www.storagecraft.com
  • Don_BurnDon_Burn Member - All Emails Posts: 1,653
    Yes WDF will support strange drivers, over the years I've done a number of
    drivers that were considered on the edge. In some ways you can think of WDF
    as analogous to object oriented programming, it can help a lot though you
    may want it in moderation.


    Don Burn
    Windows Driver Consulting
    Website: http://www.windrvr.com




    -----Original Message-----
    From: xxxxx@lists.osr.com
    [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@hotmail.com
    Sent: Monday, December 12, 2016 11:23 AM
    To: Windows System Software Devs Interest List <xxxxx@lists.osr.com>
    Subject: RE:[ntdev] Win10: crashes the Verifier with inbox-only drivers

    @Don

    The kind of windows kernel work we get in Europe isn't often the kind of
    straightforward driver work that is what WDF is aimed at, instead it is
    often weird stuff, such as allowing a driver to be unloaded even though the
    app has a handle open on it, or the CPU speed management driver that had to
    keep the CPU throttled down even though an app was running full time
    (intercepts ACPI Irps, gets the range of speeds and associated values to
    write to the CPU port, sure you remember it, it caused quite a stir when I
    discussed it here.). Or now, a USB bridge (as well as an RT KS audio
    bridge) that 'robs' a device from appearing to user mode on one system, and
    sends the data down a USB connection to another.

    Does WDF work for these kinds of *hack* drivers?

    (And from what I did read of WDF it seemed to take care of power and pnp,
    the two really tricky aspects of drivers, but now I have a large library of
    boilerplate all my code in this respect is old and working well, so are
    there other big advantages to WDF?)



    ---
    NTDEV is sponsored by OSR

    Visit the list online at:
    <http://www.osronline.com/showlists.cfm?list=ntdev>;

    MONTHLY seminars on crash dump analysis, WDF, Windows internals and software
    drivers!
    Details at <http://www.osr.com/seminars>;

    To unsubscribe, visit the List Server section of OSR Online at
    <http://www.osronline.com/page.cfm?name=ListServer>;
  • matt_sykesmatt_sykes Member - All Emails Posts: 270
    OK, I'll seriously think about migrating to it.
  • Maxim_S._ShatskihMaxim_S._Shatskih Member Posts: 10,396
    Inbox CsvVBus driver (the client-side block device driver for a CSV, it is where CsvFs is mounted) is already KMDF-based in 2012 R2.

    --
    Maxim S. Shatskih
    former Microsoft MVP on File System And Storage
    xxxxx@storagecraft.com
    http://www.storagecraft.com

    "Don Burn" <xxxxx@windrvr.com> wrote in message news:xxxxx@ntdev...
    > Yes WDF will support strange drivers, over the years I've done a number of
    > drivers that were considered on the edge. In some ways you can think of WDF
    > as analogous to object oriented programming, it can help a lot though you
    > may want it in moderation.
    >
    >
    > Don Burn
    > Windows Driver Consulting
    > Website: http://www.windrvr.com
    >
    >
    >
    >
    > -----Original Message-----
    > From: xxxxx@lists.osr.com
    > [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@hotmail.com
    > Sent: Monday, December 12, 2016 11:23 AM
    > To: Windows System Software Devs Interest List <xxxxx@lists.osr.com>
    > Subject: RE:[ntdev] Win10: crashes the Verifier with inbox-only drivers
    >
    > @Don
    >
    > The kind of windows kernel work we get in Europe isn't often the kind of
    > straightforward driver work that is what WDF is aimed at, instead it is
    > often weird stuff, such as allowing a driver to be unloaded even though the
    > app has a handle open on it, or the CPU speed management driver that had to
    > keep the CPU throttled down even though an app was running full time
    > (intercepts ACPI Irps, gets the range of speeds and associated values to
    > write to the CPU port, sure you remember it, it caused quite a stir when I
    > discussed it here.). Or now, a USB bridge (as well as an RT KS audio
    > bridge) that 'robs' a device from appearing to user mode on one system, and
    > sends the data down a USB connection to another.
    >
    > Does WDF work for these kinds of *hack* drivers?
    >
    > (And from what I did read of WDF it seemed to take care of power and pnp,
    > the two really tricky aspects of drivers, but now I have a large library of
    > boilerplate all my code in this respect is old and working well, so are
    > there other big advantages to WDF?)
    >
    >
    >
    > ---
    > NTDEV is sponsored by OSR
    >
    > Visit the list online at:
    > <http://www.osronline.com/showlists.cfm?list=ntdev>;
    >
    > MONTHLY seminars on crash dump analysis, WDF, Windows internals and software
    > drivers!
    > Details at <http://www.osr.com/seminars>;
    >
    > To unsubscribe, visit the List Server section of OSR Online at
    > <http://www.osronline.com/page.cfm?name=ListServer>;
    >
    >
  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 7,284
    <quote>
    but now I have a large library of
    boilerplate all my code in this respect is old and working well, so are there
    other big advantages to WDF
    </quote>

    TL;DR? It's worth it to move from WDM to WDF... even if you're a very experienced WDM dev with a ton of proven code. I did it back in 2005 or whenever... and I don't want to ever go back.

    The Long Version:

    Back in 2005, I was in precisely the same place that you are now. I though I had a pretty good handle on WDM, and certainly had a reasonable amount of code under my belt that I could cut/paste as required to do what I needed.

    There are several things that drove me to WDF, however:

    - It's brief: You can write a fully working driver for a piece of real hardware in 1200 or so lines of code. I challenge you to write the WDM code to handle JUST your PnP and Power requirements in that number of lines. IIRC, 1200 lines of code for PnP/Power was the MINIMUM I could get away with in WDM... and that was basically code that did nothing other than Keep Windows Happy.

    - It's mostly well thought-out and pleasant to use: It either handles for you or provides a tidy model for handling a lot of the annoying shite that one must constantly deal with in WDM. Queuing IRPs is a classic example. Ugh. I found that WDF lets me get on with worrying about my device faster, spending less time worry about Keeping Windows Happy.

    - There's less legacy stuff to drag along: With each release of Windows, I feel like in WDM there's some much old, legacy, stuff that I'm encumbered by. Good decision, bad decisions, really no matter. They're like the chain worn by Jacob Marley's Ghost in A Christmas Carol (http://paranoiastrikesdeep.blogspot.com/2013/12/i-wear-chain-i-forged-in-life.html). Remember to set FILE_DEVICE_SECURE_OPEN? Remember under which circumstances the "Exclusive" argument to IoCreateDevice actually does what it says it does? You can't avoid ALL the silliness, but you can avoid a lot of it with WDF.

    - The debugging/checking/development process is far superior by design: You really notice this when you teach a group of devs to write Windows drivers. In WDM class, the development, test, fix, re-test process usually includes a regular progression of Blue Screens. In WDF class, this same process most often avoids Blue Screens, and instead you get back some sort of error from an API that you call. Not all the time, but MUCH of the time. And when you DO get a BSOD, it's much easier to figure out what went wrong.

    I could go on all day. Hmmmm... sometimes I do. But, in any case, as a WDM developer with a reasonable amount of experience, I switched to WDF a little more than 10 years ago... and it is my goal and practice to never write a WDM driver where a WDF driver will do.

    Peter
    OSR
    @OSRDrivers

    Peter Viscarola
    OSR
    @OSRDrivers

  • matt_sykesmatt_sykes Member - All Emails Posts: 270
    You make a convincing argument, I'll start reading up on it!
Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Upcoming OSR Seminars
Developing Minifilters 29 July 2019 OSR Seminar Space
Writing WDF Drivers 23 Sept 2019 OSR Seminar Space
Kernel Debugging 21 Oct 2019 OSR Seminar Space
Internals & Software Drivers 18 Nov 2019 Dulles, VA