Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Before Posting...
Please check out the Community Guidelines in the Announcements and Administration Category.

Aaand...the PAGED code strikes yet again!!

Alex_GrigAlex_Grig Member Posts: 3,238
It's no secret Microsoft got a stick up their collective rear about PAGED code. Unfortunately, it's causing pain (ITA) to innocent bystanders, too.

We've been getting mysterious crashes while running one of WHCK (or whatever it's called at this moment) tests. It takes a few hours to happen. Here are snippets of !analyze:

The page fault happens on the instruction fetch at IRQL=2 at srv2+0x51716.

Arg1: fffff807f4361716, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000008, value 0 = read operation, 1 = write operation
Arg4: fffff807f4361716, address which referenced memory

This is PAGED code section:

SECTION HEADER #7
PAGE name
35CA7 virtual size
42000 virtual address
35E00 size of raw data
3EE00 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
60000020 flags
Code
(no align specified)
Execute Read

The code executing at the moment:

fffff807`f4361710 ff1562befeff call qword ptr [srv2+0x3d578 (fffff807`f434d578)]
fffff807`f4361716 488bcf mov rcx,rdi
fffff807`f4361719 e8a26dfbff call srv2+0x84c0 (fffff807`f43184c0)
fffff807`f436171e 488d4d07 lea rcx,[rbp+7]
fffff807`f4361722 8bd8 mov ebx,eax
fffff807`f4361724 85c0 test eax,eax
fffff807`f4361726 7969 jns srv2+0x51791 (fffff807`f4361791)
fffff807`f4361728 ff1552befeff call qword ptr [srv2+0x3d580 (fffff807`f434d580)]
fffff807`f436172e 4c8b15cb58feff mov r10,qword ptr [srv2+0x37000 (fffff807`f4347000)]

(!analyze also returns a whole bunch of red herring of !chkimg output, which is nothing else than relocated pointers. !chkimg needs to be smarter about that).

The very first call (qword ptr [srv2+0x3d578 (fffff807`f434d578)]) is (Ba Dum Tss!) nt!KeAcquireInStackQueuedSpinLock! Upon returning from it, the processor found itself at an invalid page (most likely because of Driver verifier trickery).

This is actually not the first time we caught a bugcheck because Microsoft had some code in paged section which was running at DISPATCH_LEVEL. I don't know if the driver analysis detects calls that raise IRQL in the paged code sections. Would have caught this crap at once. Or Microsoft didn't run srv2 code through analysis.

Comments

  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 7,138
    Nice, right? "Let's make everything pageable" arrrgh!

    I've got an opposite but related issue: *I* was caught yesterday by what is apparently an incorrect PAGED_CODE macro in RtlInitializeBitMap. After this code had been running for months, all of a sudden I got a break point telling me that RtlInitializeBitMap can't be called at IRQL DISPATCH_LEVEL.

    Wrong: If the bitmap and the variables are non-paged, you CAN call this function at elevated IRQL.

    Ugh. I think this was Windows 7 that suddenly complained.

    Peter
    OSR
    @OSRDrivers

    Peter Viscarola
    OSR
    @OSRDrivers

  • Alex_GrigAlex_Grig Member Posts: 3,238
    The joke is on you, because in Windows 7, RtlInitializeBitMap is in PAGE section of ntoskrnlpa.exe.

    1535 5F8 00269C84 RtlInitializeBitMap

    SECTION HEADER #8
    PAGE name
    1B24A8 virtual size
    170000 virtual address (00570000 to 007224A7)
    1B2600 size of raw data
    12CE00 file pointer to raw data (0012CE00 to 002DF3FF)
    0 file pointer to relocation table
    0 file pointer to line numbers
    0 number of relocations
    0 number of line numbers
    60000020 flags
    Code
    Execute Read


    In Windows 8+ it's in .text section.
  • Alex_GrigAlex_Grig Member Posts: 3,238
    It seems the bug is already fixed. KeAcquireInStackQueuedSpinLock call is present in srv2.sys built on Aug 23 19:42:28 2016, but not in the build of Sep 15 09:30:41 2016. There are no other IRQL-raising calls in PAGE section.
  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 7,138
    <quote>
    The joke is on you
    </quote>

    You know, it so often is.

    Sigh. I admit it: Neither I nor anyone in my office had the time or energy to look this up. We on "the long march" to getting stuff done before the holidays.

    Thank you Mr. Grig, for taking the time to explore this. That was very nice of you. I owe you one.

    I think. Because now I have to re-open the bug I closed. And figure out a way to fix this bug, and find out which other bitmap related RTL functions are similarly affected. On one historic platform. Because somebody decided this to put these routines in paged sections. To save space. In the 21st century. Arrrgh.

    Peter
    OSR
    @OSRDrivers

    Peter Viscarola
    OSR
    @OSRDrivers

  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    >...all of a sudden I got a break point telling me that RtlInitializeBitMap can't be called at IRQL DISPATCH_LEVEL.

    >... and find out which other bitmap related RTL functions are similarly affected.

    <MSDN>
    Callers of RtlInitializeBitMap, and callers of other RtlXxx routines that operate on an initialized bitmap variable, must be running at IRQL <= APC_LEVEL if the memory that contains the bitmap variable is pageable or the memory at BitmapHeader is pageable. Otherwise, RtlInitializeBitMap can be called at any IRQL.
    </MSDN>

    When a function that is marked with PAGED_CODE calls on KeAcquireInStackQueuedSpinLock, code analysis iussues warning C28150:

    "KeAcquireInStackQueuedSpinLock causes the IRQ Level to be set above the maximum acceptable for the function being analyzed: The level limits come from being PAGED_CODE or annotations on the current function."
  • Alex_GrigAlex_Grig Member Posts: 3,238
    >... and find out which other bitmap related RTL functions are similarly
    affected.

    Run dumpbin /exports on ntkrnlpa.exe, sort the exports by address (text sort by the corresponding column range should work), and select those in PAGE section address range.
  • Pavel_APavel_A Member Posts: 2,660
    Is there some registry flag that disables paging of kernel code or data? It could be used (at least) for quick test if paging related problem is suspected.

    -- pa
  • Daniel_TerhellDaniel_Terhell Member Posts: 1,349
    >Is there some registry flag that disables paging of kernel code or data? It
    >could be used (at least) for quick test if paging >related problem is
    >suspected.

    There is, DisablePagingExecutive.

    https://technet.microsoft.com/en-us/library/cc959492.aspx

    //Daniel
  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 7,138
    Because it'll make me feel better to continue to cry over spilt milk, I just wanted to confirm that RtlInitializeBitMap is pageable on Windows 7... contrary to the docs. And that's the only bitmap related function that's pageable.

    What's REALLY nice about this is the only things RtlInitializeBitMap does is assign the arguments passed into it to two fields in the BitMap header. So, you know... lots of memory is being saved by making this function pageable.

    The good news is that the work-around is simple (initialize the BitMap header manually) and easy to verify.

    Peter
    OSR
    @OSRDrivers

    Peter Viscarola
    OSR
    @OSRDrivers

  • Peter_WielandPeter_Wieland Member - All Emails Posts: 308
    Fixed in 2010, so for Windows 8 and possibly for some Win7 service pack.

    Remember you have a lot more memory now than you used to have :)

    -p

    -----Original Message-----
    From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@osr.com
    Sent: Thursday, December 8, 2016 8:39 AM
    To: Windows System Software Devs Interest List <xxxxx@lists.osr.com>
    Subject: RE:[ntdev] Aaand...the PAGED code strikes yet again!!

    Because it'll make me feel better to continue to cry over spilt milk, I just wanted to confirm that RtlInitializeBitMap is pageable on Windows 7... contrary to the docs. And that's the only bitmap related function that's pageable.

    What's REALLY nice about this is the only things RtlInitializeBitMap does is assign the arguments passed into it to two fields in the BitMap header. So, you know... lots of memory is being saved by making this function pageable.

    The good news is that the work-around is simple (initialize the BitMap header manually) and easy to verify.

    Peter
    OSR
    @OSRDrivers


    ---
    NTDEV is sponsored by OSR

    Visit the list online at: <http://www.osronline.com/showlists.cfm?list=ntdev>;

    MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
    Details at <http://www.osr.com/seminars>;

    To unsubscribe, visit the List Server section of OSR Online at <http://www.osronline.com/page.cfm?name=ListServer>;
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    Yes, __drv_maxIRQL(APC_LEVEL) is present in WDK7.1 for RtlInitializeBitmap but not in WDK8 and later.

    Interesting. A good example of why you should not run code compiled with WDK8 on W7.

    The W7 loader actually is not compatible with WDK8 binaries. The default FxDriverEntry of WDK8 crashes the machine if the stack cookie was not randomized by the loader and the W7 loader does not randomize the stack cookie.

    But I remember of blogs explaining how to load drivers built with WDK8 on W7. So now we know that this should not be done.
  • Doron_HolanDoron_Holan Member - All Emails Posts: 10,428
    It isn't FxDriverEntry that is incompatible, it is the newer BufferOverflowK.lib (with a new name, but forgot it). You can specify the older one by settings an anbuild property in the projecy, something close to $(BufferOverflowLib).

    Get Outlook for Android

    ________________________________
    From: xxxxx@lists.osr.com on behalf of xxxxx@gmail.com
    Sent: Friday, December 9, 2016 6:50:06 PM
    To: Windows System Software Devs Interest List
    Subject: RE:[ntdev] Aaand...the PAGED code strikes yet again!!

    Yes, __drv_maxIRQL(APC_LEVEL) is present in WDK7.1 for RtlInitializeBitmap but not in WDK8 and later.

    Interesting. A good example of why you should not run code compiled with WDK8 on W7.

    The W7 loader actually is not compatible with WDK8 binaries. The default FxDriverEntry of WDK8 crashes the machine if the stack cookie was not randomized by the loader and the W7 loader does not randomize the stack cookie.

    But I remember of blogs explaining how to load drivers built with WDK8 on W7. So now we know that this should not be done.

    ---
    NTDEV is sponsored by OSR

    Visit the list online at:

    MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
    Details at

    To unsubscribe, visit the List Server section of OSR Online at
    d
Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Upcoming OSR Seminars
Writing WDF Drivers 25 Feb 2019 OSR Seminar Space
Developing Minifilters 8 April 2019 OSR Seminar Space