Aaand...the PAGED code strikes yet again!!

It’s no secret Microsoft got a stick up their collective rear about PAGED code. Unfortunately, it’s causing pain (ITA) to innocent bystanders, too.

We’ve been getting mysterious crashes while running one of WHCK (or whatever it’s called at this moment) tests. It takes a few hours to happen. Here are snippets of !analyze:

The page fault happens on the instruction fetch at IRQL=2 at srv2+0x51716.

Arg1: fffff807f4361716, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000008, value 0 = read operation, 1 = write operation
Arg4: fffff807f4361716, address which referenced memory

This is PAGED code section:

SECTION HEADER #7
PAGE name
35CA7 virtual size
42000 virtual address
35E00 size of raw data
3EE00 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
60000020 flags
Code
(no align specified)
Execute Read

The code executing at the moment:

fffff807f4361710 ff1562befeff call qword ptr [srv2+0x3d578 (fffff807f434d578)]
fffff807f4361716 488bcf mov rcx,rdi fffff807f4361719 e8a26dfbff call srv2+0x84c0 (fffff807f43184c0) fffff807f436171e 488d4d07 lea rcx,[rbp+7]
fffff807f4361722 8bd8 mov ebx,eax fffff807f4361724 85c0 test eax,eax
fffff807f4361726 7969 jns srv2+0x51791 (fffff807f4361791)
fffff807f4361728 ff1552befeff call qword ptr [srv2+0x3d580 (fffff807f434d580)]
fffff807f436172e 4c8b15cb58feff mov r10,qword ptr [srv2+0x37000 (fffff807f4347000)]

(!analyze also returns a whole bunch of red herring of !chkimg output, which is nothing else than relocated pointers. !chkimg needs to be smarter about that).

The very first call (qword ptr [srv2+0x3d578 (fffff807`f434d578)]) is (Ba Dum Tss!) nt!KeAcquireInStackQueuedSpinLock! Upon returning from it, the processor found itself at an invalid page (most likely because of Driver verifier trickery).

This is actually not the first time we caught a bugcheck because Microsoft had some code in paged section which was running at DISPATCH_LEVEL. I don’t know if the driver analysis detects calls that raise IRQL in the paged code sections. Would have caught this crap at once. Or Microsoft didn’t run srv2 code through analysis.

Nice, right? “Let’s make everything pageable” arrrgh!

I’ve got an opposite but related issue: *I* was caught yesterday by what is apparently an incorrect PAGED_CODE macro in RtlInitializeBitMap. After this code had been running for months, all of a sudden I got a break point telling me that RtlInitializeBitMap can’t be called at IRQL DISPATCH_LEVEL.

Wrong: If the bitmap and the variables are non-paged, you CAN call this function at elevated IRQL.

Ugh. I think this was Windows 7 that suddenly complained.

Peter
OSR
@OSRDrivers

The joke is on you, because in Windows 7, RtlInitializeBitMap is in PAGE section of ntoskrnlpa.exe.

1535 5F8 00269C84 RtlInitializeBitMap

SECTION HEADER #8
PAGE name
1B24A8 virtual size
170000 virtual address (00570000 to 007224A7)
1B2600 size of raw data
12CE00 file pointer to raw data (0012CE00 to 002DF3FF)
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
60000020 flags
Code
Execute Read

In Windows 8+ it’s in .text section.

It seems the bug is already fixed. KeAcquireInStackQueuedSpinLock call is present in srv2.sys built on Aug 23 19:42:28 2016, but not in the build of Sep 15 09:30:41 2016. There are no other IRQL-raising calls in PAGE section.

You know, it so often is.

Sigh. I admit it: Neither I nor anyone in my office had the time or energy to look this up. We on “the long march” to getting stuff done before the holidays.

Thank you Mr. Grig, for taking the time to explore this. That was very nice of you. I owe you one.

I think. Because now I have to re-open the bug I closed. And figure out a way to fix this bug, and find out which other bitmap related RTL functions are similarly affected. On one historic platform. Because somebody decided this to put these routines in paged sections. To save space. In the 21st century. Arrrgh.

Peter
OSR
@OSRDrivers

>…all of a sudden I got a break point telling me that RtlInitializeBitMap can’t be called at IRQL DISPATCH_LEVEL.

… and find out which other bitmap related RTL functions are similarly affected.


Callers of RtlInitializeBitMap, and callers of other RtlXxx routines that operate on an initialized bitmap variable, must be running at IRQL <= APC_LEVEL if the memory that contains the bitmap variable is pageable or the memory at BitmapHeader is pageable. Otherwise, RtlInitializeBitMap can be called at any IRQL.

When a function that is marked with PAGED_CODE calls on KeAcquireInStackQueuedSpinLock, code analysis iussues warning C28150:

“KeAcquireInStackQueuedSpinLock causes the IRQ Level to be set above the maximum acceptable for the function being analyzed: The level limits come from being PAGED_CODE or annotations on the current function.”

>… and find out which other bitmap related RTL functions are similarly
affected.

Run dumpbin /exports on ntkrnlpa.exe, sort the exports by address (text sort by the corresponding column range should work), and select those in PAGE section address range.

Is there some registry flag that disables paging of kernel code or data? It could be used (at least) for quick test if paging related problem is suspected.

– pa

>Is there some registry flag that disables paging of kernel code or data? It

could be used (at least) for quick test if paging >related problem is
suspected.

There is, DisablePagingExecutive.

https://technet.microsoft.com/en-us/library/cc959492.aspx

//Daniel

Because it’ll make me feel better to continue to cry over spilt milk, I just wanted to confirm that RtlInitializeBitMap is pageable on Windows 7… contrary to the docs. And that’s the only bitmap related function that’s pageable.

What’s REALLY nice about this is the only things RtlInitializeBitMap does is assign the arguments passed into it to two fields in the BitMap header. So, you know… lots of memory is being saved by making this function pageable.

The good news is that the work-around is simple (initialize the BitMap header manually) and easy to verify.

Peter
OSR
@OSRDrivers

Fixed in 2010, so for Windows 8 and possibly for some Win7 service pack.

Remember you have a lot more memory now than you used to have :slight_smile:

-p

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@osr.com
Sent: Thursday, December 8, 2016 8:39 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Aaand…the PAGED code strikes yet again!!

Because it’ll make me feel better to continue to cry over spilt milk, I just wanted to confirm that RtlInitializeBitMap is pageable on Windows 7… contrary to the docs. And that’s the only bitmap related function that’s pageable.

What’s REALLY nice about this is the only things RtlInitializeBitMap does is assign the arguments passed into it to two fields in the BitMap header. So, you know… lots of memory is being saved by making this function pageable.

The good news is that the work-around is simple (initialize the BitMap header manually) and easy to verify.

Peter
OSR
@OSRDrivers


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:>

Yes, __drv_maxIRQL(APC_LEVEL) is present in WDK7.1 for RtlInitializeBitmap but not in WDK8 and later.

Interesting. A good example of why you should not run code compiled with WDK8 on W7.

The W7 loader actually is not compatible with WDK8 binaries. The default FxDriverEntry of WDK8 crashes the machine if the stack cookie was not randomized by the loader and the W7 loader does not randomize the stack cookie.

But I remember of blogs explaining how to load drivers built with WDK8 on W7. So now we know that this should not be done.

It isn’t FxDriverEntry that is incompatible, it is the newer BufferOverflowK.lib (with a new name, but forgot it). You can specify the older one by settings an anbuild property in the projecy, something close to $(BufferOverflowLib).

Get Outlook for Android


From: xxxxx@lists.osr.com on behalf of xxxxx@gmail.com
Sent: Friday, December 9, 2016 6:50:06 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Aaand…the PAGED code strikes yet again!!

Yes, __drv_maxIRQL(APC_LEVEL) is present in WDK7.1 for RtlInitializeBitmap but not in WDK8 and later.

Interesting. A good example of why you should not run code compiled with WDK8 on W7.

The W7 loader actually is not compatible with WDK8 binaries. The default FxDriverEntry of WDK8 crashes the machine if the stack cookie was not randomized by the loader and the W7 loader does not randomize the stack cookie.

But I remember of blogs explaining how to load drivers built with WDK8 on W7. So now we know that this should not be done.


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:>