Paged or Non Paged ???

[OSR_Community_User]

Yep, this is a deep concept. And perhaps NT insider should have an article
on it !!!.

The interplay of memory with process virtual address space, working set
etc., are to be spelled out in bit more detail …

For example -

(1) Who creates a VA? What does it consists of? In a VA what kind of things
could happen (a) I access some address I get the result immediately (b) I
access and get a page fault (c) I access and I get an access violations …

(2) For executable image section ( from: exe, dll, sys ) who is/are the
backing store? Is it page file or image file

(3) What buys due to this mapping ?

(4) Pools and Non pools

(5) How the memory is crafted

etc., etc.

-pro

On Tue, Apr 28, 2009 at 7:46 PM, Alex Shvedov wrote:

> Well you could trust the wise old folks who are telling you it really
>> exists
>> trusting Google
>>
> Trust? It was not and is not a matter of trust.
>
> If you are not prepared to believe the experts who know the Windows
>> source code intimately
>>
> Believe? Well…
> I in fact asked “what are memory regions not under mm’s control”.
> There is no place for trust or belief here, the table (which
> I in fact simply forgot) I googled out provided the answer.
>
> Jic this does not mean that I believe/trust (or love or hate etc.) the
> table.
>
> your only option is to find some way to look at the source code yourself.
>>
> I did not have this option at the time. As it turns out, google may help,
> just
> do not look at the ads.
>
> There are three levels of memory management.
>>
> Nice, useful (and correct) answer, but to a different question, the one I
> did not ask.
> Never hurts to have this memo handy anyway.
>
> Well you could use the debugger and ask it which pool an address from
>> your driver’s code or data segments is in.
>>
> As you yourself note, it is not that simple. Besides, imagine that a
> particular
> driver is in such-n-such seg, what about ntos per se? non-driver mappings?
> etc. etc. I would have to trust/believe in order to generalize…
>
> Anyway, Xpaged pool is in fact a proper subset of Xpaged memory,
> not that I trust or believe in this, I know (remembered?) now.
>
> I trust/believe that you know the diff.
>
>
>
> ----- Original Message ----- From: “Peter Wieland” <
> xxxxx@windows.microsoft.com>
> To: “Windows System Software Devs Interest List”
> Sent: Tuesday, April 28, 2009 5:42 PM
> Subject: RE: Re:[ntdev] Paged or Non Paged ???
>
>
>
> Well you could trust the wise old folks who are telling you it really
> exists. That might be more sensible than trusting Google which knows
> something about everything but is an expert in nothing (except perhaps
> advertising :)). You could also buy a copy of the Windows Internals book
> which explains a lot of this.
>
> There are three levels of memory management.
>
> At one level MM manages physical pages of memory. MM keeps track of
> whether each physical page is in use, free, zeroed, locked, dirty, etc…
>
> Above that is MM’s management of the virtual address space (VA) -
> per-process VA spaces, per-session VA space and the kernel VA space. MM
> manages the reservation and release of VA ranges and separately tracks which
> physical page is assigned to each virtual page. At this level MM manages
> which virtual addresses are being used in which address spaces and can allow
> their corresponding physical pages to be brought into or removed from the
> process’s working set.
>
> When a physical page is evicted from a working set (either by being freed
> or being aged out) that drops its locked count. When the lock count drops
> to zero the physical page can be freed. If the page is dirty and backed by
> a file then MM will write the contents out to the file before freeing it.
> If the VA was evicted and is accessed by the process then a page fault
> occurs and MM will allocate a new physical page and read the appropriate
> contents back in from the backing file.
>
> The equivalent of this in user-mode is VirtualAlloc, which allows an
> application to reserve a chunk of its own VA space and also to back that
> chunk by physical memory. Similarly CreateFileMapping allows the process to
> allocate a virtual address range which is backed by a file. When MM goes to
> make a page in that chunk resident it will load the contents from the
> backing file rather than just using a zeroed page (as it would with
> VirtualAlloc).
>
> Pool is built on top of this mechanism. It’s the equivalent of the
> user-mode heap or the CRT’s malloc function (not the equivalent of a malloc
> system call). The pool allocates virtual memory from Mm and then subdivides
> that into individual pool allocations for the callers. Pool is just one way
> of getting virtual address space, and one of the only options exported by
> the kernel, but it is separate from the underlying virtual memory management
> layer that it depends on.
>
> -p
>
> -----Original Message-----
> From: xxxxx@lists.osr.com [mailto:
> xxxxx@lists.osr.com] On Behalf Of Alex Shvedov
> Sent: Tuesday, April 28, 2009 1:50 PM
> To: Windows System Software Devs Interest List
> Subject: Re: Re:[ntdev] Paged or Non Paged ???
>
> system PTEs are allocated to map the driver image, the driver image is
>> copied into memory
>>
> What kind of memory?
> ExAllocated by driver loader?
>
> common newb mistake is to use the terms “paged pool” and “non-page
>> pool” interchangeably with “pageable system memory” and “non-pageable
>> system memory”
>>
> Would it be too much to ask you to explain the difference?
> Googling brought up nothing, really - except an obvious fact that
> [non]pageable pool belongs to [non]pageable memory, like in
> WDK’s POOL_TYPE description: “… Nonpaged pool, which is
> nonpageable system memory”.
>
> Not much.
>
> Is there any way to confirm that this difference that
> newbies forget about really exisst?
>
> I don’t have Mark’s bible at hand…
>
>
>
>
> ----- Original Message -----
> From: “Scott Noone”
> Newsgroups: ntdev
> To: “Windows System Software Devs Interest List”
> Sent: Tuesday, April 28, 2009 4:24 PM
> Subject: Re:[ntdev] Paged or Non Paged ???
>
>
> >I think the loader reads the whole driver from .SYS file to the paged
>> >pool area and then locks all sections whose names do not start >with
>> >“PAGE”.
>>
>> Don’t forget that drivers can also go in session space.
>>
>> And AFAIK it’s the other way around, system PTEs are allocated to map the
>> driver image, the driver image is copied into memory, and PTEs for PAGE
>> sections are marked pageable.
>>
>>
>> -scott
>>
>> –
>> Scott Noone
>> Consulting Associate
>> OSR Open Systems Resources, Inc.
>> http://www.osronline.com
>>
>>
>> “Maxim S. Shatskih” wrote in message
>> news:xxxxx@ntdev…
>>
>>> When you say that it is “mapped” in to memory…where the “memory” comes
>>> from. Doesn’t the loader
>>> has to read the .sys file from the disk and bring it in to SOME memory?
>>>
>>
>> I think the loader reads the whole driver from .SYS file to the paged pool
>> area and then locks all sections whose names do not start with “PAGE”.
>>
>> At least you can delete and replace the loaded .SYS (not so with user-mode
>> DLLs and EXEs).
>>
>> –
>> Maxim S. Shatskih
>> Windows DDK MVP
>> xxxxx@storagecraft.com
>> http://www.storagecraft.com
>>
>>
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

[N_Madhavan]

i am very much thankfull to all of you.

i am very sorry for asking a wrong question.

I think, my question can be changed to , a global variable in a driver will use pageable system memory or non-pageable system memory ? (I allready got the answer from you :- ie non-pageable system memory).
But i think it will be good if i explain why i asked this question.i have driver where i declare a static stucture in global scope.that static structure contains some HANDLEs (i know using global variable in driver is a bad practice) like

static struct
{
//some code
HANDLE write_thread ;
HANDLE timer_thread ;
KEVENT write_event ;
KEVENT timer_event;
//some code
} g_queue ;

i am using this g_queue in some functions.

The problem i am getting is, driver is getting crashed in some systems(not every time).i am using XP SP2 and XP SP3 systems.i tested in arround 10 systems.its getting crashed only in 3 systems(not every time).so i done a crash dump analysis and i got the following result

//////////debugger output start

Use !analyze -v to get detailed debugging information.

BugCheck 1000000A, {20, 2, 1, 806e684a}

Probably caused by : driver.sys ( driver!logger_thread+38 )

Followup: MachineOwner

0: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000020, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000001, bitfield :
bit 0 : value 0 = read operation, 1 = write operation
bit 3 : value 0 = not an execute operation, 1 = execute operation (only on chips which support this level of status)
Arg4: 806e684a, address which referenced memory

Debugging Details:

WRITE_ADDRESS: 00000020

CURRENT_IRQL: 2

FAULTING_IP:
hal!KfAcquireSpinLock+1a
806e684a f00fba2900 lock bts dword ptr [ecx],0

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xA

PROCESS_NAME: System

LAST_CONTROL_TRANSFER: from ba614b2e to 806e684a

STACK_TEXT:
a87c5d74 ba614b2e 00000000 8a76fb30 00000000 hal!KfAcquireSpinLock+0x1a
a87c5dac 805cff70 00000000 00000000 00000000 driver!logger_thread+0x38 [e\log.c @ 302]
a87c5ddc 805460ee ba614af6 00000000 00000000 nt!PspSystemThreadStartup+0x34
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

STACK_COMMAND: kb

FOLLOWUP_IP:
driver!logger_thread+38 [\log.c @ 302]
ba614b2e 8845ff mov byte ptr [ebp-1],al

FAULTING_SOURCE_CODE:
298: has_request = 0 ;
299:
300: // enter DISPATCH level
301: //

302: KeAcquireSpinLock(&g_queue.guard, &irql) ;
303:
304: if (g_queue.head != g_queue.tail)
//some code
KeReleaseSpinLock(&g_queue.guard, irql) ;

SYMBOL_STACK_INDEX: 1

SYMBOL_NAME: driver!logger_thread+38

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: driver

IMAGE_NAME: driver.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 49efe18d

FAILURE_BUCKET_ID: 0xA_driver!logger_thread+38

BUCKET_ID: 0xA_driver!logger_thread+38

Followup: MachineOwner

////////////////////////debuger output ends

so from debugger output

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.

so i thought ,error is because global variable is paged out from the memory.so i just want to confirm it from experts that a global variable in a driver will use pageable system memory or non-pageable system memory .

Thanks and Regards

Navaneeth

[OSR_Community_User]

> And perhaps NT insider should have an article on it !!!.
… and the reply (Peter’s, IIRC) I mentioned can be a good
starting point.

Among other things, what I found, much to my surprise, is
that WDK docs do not have a notion of, say, “nonpaged
memory” (I mean, as opposed to “nonpaged pool”, its
proper subset). Some responses therefore put WDK docs
authors in a newb category…

Of course, there is Mark’s bible, but a concise memo
[the one you can trust and believe :-)] would be really
good.

----- Original Message -----
From: Prokash Sinha
To: Windows System Software Devs Interest List
Sent: Tuesday, April 28, 2009 11:06 PM
Subject: Re: Re:[ntdev] Paged or Non Paged ???

Yep, this is a deep concept. And perhaps NT insider should have an article on it !!!.

The interplay of memory with process virtual address space, working set etc., are to be spelled out in bit more detail …

For example -

(1) Who creates a VA? What does it consists of? In a VA what kind of things could happen (a) I access some address I get the result immediately (b) I access and get a page fault (c) I access and I get an access violations …

(2) For executable image section ( from: exe, dll, sys ) who is/are the backing store? Is it page file or image file

(3) What buys due to this mapping ?

(4) Pools and Non pools

(5) How the memory is crafted

etc., etc.

-pro

On Tue, Apr 28, 2009 at 7:46 PM, Alex Shvedov wrote:

Well you could trust the wise old folks who are telling you it really exists

trusting Google

Trust? It was not and is not a matter of trust.

If you are not prepared to believe the experts who know the Windows
source code intimately

Believe? Well…
I in fact asked “what are memory regions not under mm’s control”.
There is no place for trust or belief here, the table (which
I in fact simply forgot) I googled out provided the answer.

Jic this does not mean that I believe/trust (or love or hate etc.) the table.

your only option is to find some way to look at the source code yourself.

I did not have this option at the time. As it turns out, google may help, just
do not look at the ads.

There are three levels of memory management.

Nice, useful (and correct) answer, but to a different question, the one I did not ask.
Never hurts to have this memo handy anyway.

Well you could use the debugger and ask it which pool an address from
your driver’s code or data segments is in.

As you yourself note, it is not that simple. Besides, imagine that a particular
driver is in such-n-such seg, what about ntos per se? non-driver mappings?
etc. etc. I would have to trust/believe in order to generalize…

Anyway, Xpaged pool is in fact a proper subset of Xpaged memory,
not that I trust or believe in this, I know (remembered?) now.

I trust/believe that you know the diff.

----- Original Message ----- From: “Peter Wieland”

To: “Windows System Software Devs Interest List”

Sent: Tuesday, April 28, 2009 5:42 PM
Subject: RE: Re:[ntdev] Paged or Non Paged ???

Well you could trust the wise old folks who are telling you it really exists. That might be more sensible than trusting Google which knows something about everything but is an expert in nothing (except perhaps advertising :)). You could also buy a copy of the Windows Internals book which explains a lot of this.

There are three levels of memory management.

At one level MM manages physical pages of memory. MM keeps track of whether each physical page is in use, free, zeroed, locked, dirty, etc…

Above that is MM’s management of the virtual address space (VA) - per-process VA spaces, per-session VA space and the kernel VA space. MM manages the reservation and release of VA ranges and separately tracks which physical page is assigned to each virtual page. At this level MM manages which virtual addresses are being used in which address spaces and can allow their corresponding physical pages to be brought into or removed from the process’s working set.

When a physical page is evicted from a working set (either by being freed or being aged out) that drops its locked count. When the lock count drops to zero the physical page can be freed. If the page is dirty and backed by a file then MM will write the contents out to the file before freeing it. If the VA was evicted and is accessed by the process then a page fault occurs and MM will allocate a new physical page and read the appropriate contents back in from the backing file.

The equivalent of this in user-mode is VirtualAlloc, which allows an application to reserve a chunk of its own VA space and also to back that chunk by physical memory. Similarly CreateFileMapping allows the process to allocate a virtual address range which is backed by a file. When MM goes to make a page in that chunk resident it will load the contents from the backing file rather than just using a zeroed page (as it would with VirtualAlloc).

Pool is built on top of this mechanism. It’s the equivalent of the user-mode heap or the CRT’s malloc function (not the equivalent of a malloc system call). The pool allocates virtual memory from Mm and then subdivides that into individual pool allocations for the callers. Pool is just one way of getting virtual address space, and one of the only options exported by the kernel, but it is separate from the underlying virtual memory management layer that it depends on.

-p

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Alex Shvedov
Sent: Tuesday, April 28, 2009 1:50 PM
To: Windows System Software Devs Interest List
Subject: Re: Re:[ntdev] Paged or Non Paged ???

system PTEs are allocated to map the driver image, the driver image is
copied into memory

What kind of memory?
ExAllocated by driver loader?

common newb mistake is to use the terms “paged pool” and “non-page
pool” interchangeably with “pageable system memory” and “non-pageable
system memory”

Would it be too much to ask you to explain the difference?
Googling brought up nothing, really - except an obvious fact that
[non]pageable pool belongs to [non]pageable memory, like in
WDK’s POOL_TYPE description: “… Nonpaged pool, which is
nonpageable system memory”.

Not much.

Is there any way to confirm that this difference that
newbies forget about really exisst?

I don’t have Mark’s bible at hand…

----- Original Message -----
From: “Scott Noone”
Newsgroups: ntdev
To: “Windows System Software Devs Interest List”
Sent: Tuesday, April 28, 2009 4:24 PM
Subject: Re:[ntdev] Paged or Non Paged ???

>I think the loader reads the whole driver from .SYS file to the paged
>pool area and then locks all sections whose names do not start >with
>“PAGE”.

Don’t forget that drivers can also go in session space.

And AFAIK it’s the other way around, system PTEs are allocated to map the
driver image, the driver image is copied into memory, and PTEs for PAGE
sections are marked pageable.

-scott

–
Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

“Maxim S. Shatskih” wrote in message
news:xxxxx@ntdev…

When you say that it is “mapped” in to memory…where the “memory” comes
from. Doesn’t the loader
has to read the .sys file from the disk and bring it in to SOME memory?

I think the loader reads the whole driver from .SYS file to the paged pool
area and then locks all sections whose names do not start with “PAGE”.

At least you can delete and replace the loaded .SYS (not so with user-mode
DLLs and EXEs).

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

[Mark_Roddy]

On Wed, Apr 29, 2009 at 2:53 AM, wrote:
> STACK_TEXT:
> a87c5d74 ba614b2e 00000000 8a76fb30 00000000 hal!KfAcquireSpinLock+0x1a
> a87c5dac 805cff70 00000000 00000000 00000000 driver!logger_thread+0x38 [e\log.c @ 302]

The address passed to KeAcquireSpinLock for the lock appears to be
NULL. Offhand how your code sample does that is not clear to me.

The debugger would be happy to output the current values of driver!
g_queue at the time of the crash. “dt driver!g_queue” ought to work.

Is it possible that you have redefined this struct? You omitted the
definition of the spinlock in g_queue, is it defined as a PKSPIN_LOCK
or a KSPIN_LOCK? Have you run your code through prefast? Have you run
your driver with driver verifier enabled? How do you handle thread
cleanup on driver unload, if you support unload? What provokes this
crash?

Mark Roddy

[N_Madhavan]

static struct
{
PNACSLog data ;
KSPIN_LOCK guard ;
ULONG head ;
ULONG tail ;
HANDLE file ;
HANDLE write_thread ;
HANDLE timer_thread ;
KEVENT write_event ;
KEVENT timer_event;
BOOLEAN b_exit ;
} g_queue ;

is the complete structure i used

i used KSPIN_LOCK.
i did not use prefast and driver verifier.

Crash was occured while loading the driver( but not allways ,only some time).

i initialize spinlock in DriverEntry by calling
KeInitializeSpinLock(&g_queue.guard) ;

then create the thread by calling

status = PsCreateSystemThread(&g_queue.write_thread,
THREAD_ALL_ACCESS,
NULL,
NULL,
NULL,
logger_thread,
NULL) ;

[Mark_Roddy]

Run prefast, fix all the real errors it exposes and then test against
driver verifier. Does your driver support unload?

Mark Roddy

On Wed, Apr 29, 2009 at 9:53 AM, wrote:
> static struct
> {
> ? ? ? ?PNACSLog ? ?data ;
> ? ? ? ?KSPIN_LOCK ? ? ?guard ;
> ? ? ? ?ULONG ? ? ? ? ? head ;
> ? ? ? ?ULONG ? ? ? ? ? tail ;
> ? ? ? ?HANDLE ? ? ? ? ?file ;
> ? ? ? ?HANDLE ? ? ? ? ?write_thread ;
> ? ? ? ?HANDLE ? ? ? ? ?timer_thread ;
> ? ? ? ?KEVENT ? ? ? ? ?write_event ;
> ? ? ? ?KEVENT ? ? ? ? ?timer_event;
> ? ? ? ?BOOLEAN ? ? ? ? b_exit ;
> } g_queue ;
>
> is the complete structure i used
>
> i used KSPIN_LOCK.
> i did not use prefast and driver verifier.
>
> Crash was occured while loading the driver( but not allways ,only some time).
>
> i initialize spinlock in DriverEntry by calling
> KeInitializeSpinLock(&g_queue.guard) ;
>
> then create the thread by calling
>
> status = PsCreateSystemThread(&g_queue.write_thread,
> ? ? ? ? ? ? ? ?THREAD_ALL_ACCESS,
> ? ? ? ? ? ? ? ?NULL,
> ? ? ? ? ? ? ? ?NULL,
> ? ? ? ? ? ? ? ?NULL,
> ? ? ? ? ? ? ? ?logger_thread,
> ? ? ? ? ? ? ? ?NULL) ;
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>

[Ken_Johnson]

KfAcquireSpinLock is _

• S

-----Original Message-----
From: Mark Roddy
Sent: Wednesday, April 29, 2009 06:22
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Paged or Non Paged ???

On Wed, Apr 29, 2009 at 2:53 AM, wrote:
> STACK_TEXT:
> a87c5d74 ba614b2e 00000000 8a76fb30 00000000 hal!KfAcquireSpinLock+0x1a
> a87c5dac 805cff70 00000000 00000000 00000000 driver!logger_thread+0x38 [e\log.c @ 302]

The address passed to KeAcquireSpinLock for the lock appears to be
NULL. Offhand how your code sample does that is not clear to me.

The debugger would be happy to output the current values of driver!
g_queue at the time of the crash. “dt driver!g_queue” ought to work.

Is it possible that you have redefined this struct? You omitted the
definition of the spinlock in g_queue, is it defined as a PKSPIN_LOCK
or a KSPIN_LOCK? Have you run your code through prefast? Have you run
your driver with driver verifier enabled? How do you handle thread
cleanup on driver unload, if you support unload? What provokes this
crash?

Mark Roddy

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

[Ken_Johnson]

KfAcquireSpinLock is __fastcall on x86 and the argument would not show on a “kv” stack dump as it is by-register (ecx).

• S

-----Original Message-----
From: Mark Roddy
Sent: Wednesday, April 29, 2009 06:22
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Paged or Non Paged ???

On Wed, Apr 29, 2009 at 2:53 AM, wrote:
> STACK_TEXT:
> a87c5d74 ba614b2e 00000000 8a76fb30 00000000 hal!KfAcquireSpinLock+0x1a
> a87c5dac 805cff70 00000000 00000000 00000000 driver!logger_thread+0x38 [e\log.c @ 302]

The address passed to KeAcquireSpinLock for the lock appears to be
NULL. Offhand how your code sample does that is not clear to me.

The debugger would be happy to output the current values of driver!
g_queue at the time of the crash. “dt driver!g_queue” ought to work.

Is it possible that you have redefined this struct? You omitted the
definition of the spinlock in g_queue, is it defined as a PKSPIN_LOCK
or a KSPIN_LOCK? Have you run your code through prefast? Have you run
your driver with driver verifier enabled? How do you handle thread
cleanup on driver unload, if you support unload? What provokes this
crash?

Mark Roddy

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

[Scott_Noone_OSR]

His bugcheck shows that it’s likely the spinlock address he passed in was
somehow 0x20:

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at
an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000020, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000001, bitfield :

FAULTING_IP:
hal!KfAcquireSpinLock+1a
806e684a f00fba2900 lock bts dword ptr [ecx],0

Which seems like it might be the field offset (full structure wasn’t pasted
so hard to say). Though I have no idea how that happened based on the code
shown.

-scott

–
Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

“Skywing” wrote in message
news:xxxxx@ntdev…
KfAcquireSpinLock is __fastcall on x86 and the argument would not show on a
“kv” stack dump as it is by-register (ecx).

- S

-----Original Message-----
From: Mark Roddy
Sent: Wednesday, April 29, 2009 06:22
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Paged or Non Paged ???

On Wed, Apr 29, 2009 at 2:53 AM, wrote:
> STACK_TEXT:
> a87c5d74 ba614b2e 00000000 8a76fb30 00000000 hal!KfAcquireSpinLock+0x1a
> a87c5dac 805cff70 00000000 00000000 00000000 driver!logger_thread+0x38
> [e\log.c @ 302]

The address passed to KeAcquireSpinLock for the lock appears to be
NULL. Offhand how your code sample does that is not clear to me.

The debugger would be happy to output the current values of driver!
g_queue at the time of the crash. “dt driver!g_queue” ought to work.

Is it possible that you have redefined this struct? You omitted the
definition of the spinlock in g_queue, is it defined as a PKSPIN_LOCK
or a KSPIN_LOCK? Have you run your code through prefast? Have you run
your driver with driver verifier enabled? How do you handle thread
cleanup on driver unload, if you support unload? What provokes this
crash?

Mark Roddy

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

[Tim_Roberts]

xxxxx@aol.in wrote:

static struct
{
PNACSLog data ;
KSPIN_LOCK guard ;
ULONG head ;
ULONG tail ;
HANDLE file ;
HANDLE write_thread ;
HANDLE timer_thread ;
KEVENT write_event ;
KEVENT timer_event;
BOOLEAN b_exit ;
} g_queue ;

is the complete structure i used

OK, since we’re playing “guess the issue”, here’s my guess, My guess
is that your PNACSLog structure is some kind of a buffer that contains
log messages, and you are writing off the end of that structure and
trashing the KSPIN_LOCK.

Why don’t you show us the definition of PNACSLog?

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

[OSR_Community_User]

> Arg1: 00000020, memory referenced
This tells me that you are trying to access a member at offset 0x20
within a struct which is NULL.
Something is NULL and it is assumed not to be.

-----Original Message-----
From: Scott Noone [mailto:xxxxx@osr.com]
Sent: Wednesday, April 29, 2009 9:51 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Paged or Non Paged ???

His bugcheck shows that it’s likely the spinlock address he passed in
was
somehow 0x20:

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address
at
an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000020, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000001, bitfield :

FAULTING_IP:
hal!KfAcquireSpinLock+1a
806e684a f00fba2900 lock bts dword ptr [ecx],0

Which seems like it might be the field offset (full structure wasn’t
pasted
so hard to say). Though I have no idea how that happened based on the
code
shown.

-scott

–
Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

“Skywing” wrote in message
news:xxxxx@ntdev…
KfAcquireSpinLock is __fastcall on x86 and the argument would not show
on a
“kv” stack dump as it is by-register (ecx).

- S

-----Original Message-----
From: Mark Roddy
Sent: Wednesday, April 29, 2009 06:22
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Paged or Non Paged ???

On Wed, Apr 29, 2009 at 2:53 AM, wrote:
> STACK_TEXT:
> a87c5d74 ba614b2e 00000000 8a76fb30 00000000
hal!KfAcquireSpinLock+0x1a
> a87c5dac 805cff70 00000000 00000000 00000000 driver!logger_thread+0x38

> [e\log.c @ 302]

The address passed to KeAcquireSpinLock for the lock appears to be
NULL. Offhand how your code sample does that is not clear to me.

The debugger would be happy to output the current values of driver!
g_queue at the time of the crash. “dt driver!g_queue” ought to work.

Is it possible that you have redefined this struct? You omitted the
definition of the spinlock in g_queue, is it defined as a PKSPIN_LOCK
or a KSPIN_LOCK? Have you run your code through prefast? Have you run
your driver with driver verifier enabled? How do you handle thread
cleanup on driver unload, if you support unload? What provokes this
crash?

Mark Roddy

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

[Peter_Viscarola_OSR]

Among other things, what I found, much to my surprise, is that WDK docs do not have a notion of, say, “nonpaged memory” (I mean, as opposed to “nonpaged pool”, its proper subset). Some responses therefore put WDK docs authors in a newb category…

Who EVER said that the contents of the WDK docs were definitive?

In my experience, most (not all, but most) of the WDK doc authors have never written a driver. You can decide for yourself if that qualifies them for “newb” status. Note I didn’t say they’re not smart people, or good writers. But, there’s a lot to be said for knowing how to actually DO what you’re writing about…

Peter
OSR

[Maxim_S_Shatskih]

>Read something about NT file ( including image) mapping …

User and kernel mode image mapping do differs a lot.

User mapped image holds the file open, and pages in the pages from the image file itself on faults.

Kernel driver does not hold the .SYS file open, and pages the pages in from the pagefile on faults.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

[OSR_Community_User]

Yes it is true that sys file goes to pagefile …
Now the most non-obvious question is why they are jumbled together into
pagefile ???

If I’ve to guess, then (1) because they are relatively small in size (2)
security (3) what else ???

There are bits and pieces of informations here or there about this whole
business of ( image format, in core process structure, who loads, paged vs.
non-paged, privileges and all sorts )…

In the past I’ve seen two articles on Mm & VA on insider - nicely written,
could be nicer for newb ( and very very oldb, like me :). Something that
would match with the general concepts one gets out of college ( and does not
delve into it on a daily basis, but sometime need to get on with it ). This
is exactly the poor-soul-from-the-driver-land needs !!!

-pro

On Wed, Apr 29, 2009 at 1:56 PM, Maxim S. Shatskih
wrote:

> >Read something about NT file ( including image) mapping …
>
> User and kernel mode image mapping do differs a lot.
>
> User mapped image holds the file open, and pages in the pages from the
> image file itself on faults.
>
> Kernel driver does not hold the .SYS file open, and pages the pages in from
> the pagefile on faults.
>
> –
> Maxim S. Shatskih
> Windows DDK MVP
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

[Jake_Oshins]

The answer is simpler than you might expect. The memory manager can only
page code from its original image if it knows where that image is. Boot
drivers (which are the ones which are paged to the page file) are put into
memory by the bootloader. Thus the memory manager has no idea where they
came from on disk.

–
Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group

This post implies no warranties and confers no rights.

---

“Prokash Sinha” wrote in message news:xxxxx@ntdev…
Yes it is true that sys file goes to pagefile …

Now the most non-obvious question is why they are jumbled together into
pagefile ???

If I’ve to guess, then (1) because they are relatively small in size (2)
security (3) what else ???

There are bits and pieces of informations here or there about this whole
business of ( image format, in core process structure, who loads, paged vs.
non-paged, privileges and all sorts )…

In the past I’ve seen two articles on Mm & VA on insider - nicely written,
could be nicer for newb ( and very very oldb, like me :). Something that
would match with the general concepts one gets out of college ( and does not
delve into it on a daily basis, but sometime need to get on with it ). This
is exactly the poor-soul-from-the-driver-land needs !!!

-pro

On Wed, Apr 29, 2009 at 1:56 PM, Maxim S. Shatskih
wrote:

>Read something about NT file ( including image) mapping …

User and kernel mode image mapping do differs a lot.

User mapped image holds the file open, and pages in the pages from the image
file itself on faults.

Kernel driver does not hold the .SYS file open, and pages the pages in from
the pagefile on faults.

–

Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

—

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

[OSR_Community_User]

Thanks a bunch… That clarifies stuff.
From your stmt, Jake, now I assume that any driver that are loaded much
later in the system bring up ( meaning the Mm is active now ) phase are
indeed not using pagefile !!!.

-pro

On Wed, Apr 29, 2009 at 4:24 PM, Jake Oshins wrote:

> The answer is simpler than you might expect. The memory manager can only
> page code from its original image if it knows where that image is. Boot
> drivers (which are the ones which are paged to the page file) are put into
> memory by the bootloader. Thus the memory manager has no idea where they
> came from on disk.
>
> –
> Jake Oshins
> Hyper-V I/O Architect
> Windows Kernel Group
>
> This post implies no warranties and confers no rights.
>
> --------------------------------------------------------------
>
> “Prokash Sinha” wrote in message news:xxxxx@ntdev…
> Yes it is true that sys file goes to pagefile …
>
>
> Now the most non-obvious question is why they are jumbled together into
> pagefile ???
>
>
> If I’ve to guess, then (1) because they are relatively small in size (2)
> security (3) what else ???
>
>
> There are bits and pieces of informations here or there about this whole
> business of ( image format, in core process structure, who loads, paged vs.
> non-paged, privileges and all sorts )…
>
>
> In the past I’ve seen two articles on Mm & VA on insider - nicely written,
> could be nicer for newb ( and very very oldb, like me :). Something that
> would match with the general concepts one gets out of college ( and does not
> delve into it on a daily basis, but sometime need to get on with it ). This
> is exactly the poor-soul-from-the-driver-land needs !!!
>
>
> -pro
>
>
> On Wed, Apr 29, 2009 at 1:56 PM, Maxim S. Shatskih
> wrote:
>
> Read something about NT file ( including image) mapping …
>>
>
>
> User and kernel mode image mapping do differs a lot.
>
> User mapped image holds the file open, and pages in the pages from the
> image file itself on faults.
>
> Kernel driver does not hold the .SYS file open, and pages the pages in from
> the pagefile on faults.
>
> –
>
> Maxim S. Shatskih
> Windows DDK MVP
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> —
>
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

[OSR_Community_User]

You assume incorrectly. Drivers are backed by the page file, not the original on-disk file, no matter when they?re loaded.

-p

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Prokash Sinha
Sent: Wednesday, April 29, 2009 4:51 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Paged or Non Paged ???

Thanks a bunch… That clarifies stuff.

From your stmt, Jake, now I assume that any driver that are loaded much later in the system bring up ( meaning the Mm is active now ) phase are indeed not using pagefile !!!.

-pro
On Wed, Apr 29, 2009 at 4:24 PM, Jake Oshins > wrote:
The answer is simpler than you might expect. The memory manager can only page code from its original image if it knows where that image is. Boot drivers (which are the ones which are paged to the page file) are put into memory by the bootloader. Thus the memory manager has no idea where they came from on disk.

–
Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group

This post implies no warranties and confers no rights.

--------------------------------------------------------------

“Prokash Sinha” > wrote in message news:xxxxx@ntdev…

Yes it is true that sys file goes to pagefile …

Now the most non-obvious question is why they are jumbled together into pagefile ???

If I’ve to guess, then (1) because they are relatively small in size (2) security (3) what else ???

There are bits and pieces of informations here or there about this whole business of ( image format, in core process structure, who loads, paged vs. non-paged, privileges and all sorts )…

In the past I’ve seen two articles on Mm & VA on insider - nicely written, could be nicer for newb ( and very very oldb, like me :). Something that would match with the general concepts one gets out of college ( and does not delve into it on a daily basis, but sometime need to get on with it ). This is exactly the poor-soul-from-the-driver-land needs !!!

-pro

On Wed, Apr 29, 2009 at 1:56 PM, Maxim S. Shatskih > wrote:
Read something about NT file ( including image) mapping …

User and kernel mode image mapping do differs a lot.

User mapped image holds the file open, and pages in the pages from the image file itself on faults.

Kernel driver does not hold the .SYS file open, and pages the pages in from the pagefile on faults.

–

Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.commailto:xxxxx
http://www.storagecraft.com

—

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx>

[Prokash_Sinha-1]

Thanks Peter. Seems like I did not parse well Jake’s stmt :-).

So just an arbit question, for sys files that are loaded after Mm is
active, what are the other reasons that they have to be paged out ( if
pageable sections presents) to pagefile. Or is it just a mere
convenience to have them all use the pagefile as backing store ?

-pro

Peter Wieland wrote:

You assume incorrectly. Drivers are backed by the page file, not the
original on-disk file, no matter when they’re loaded.

-p

*From:* xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] *On Behalf Of *Prokash Sinha
*Sent:* Wednesday, April 29, 2009 4:51 PM
*To:* Windows System Software Devs Interest List
*Subject:* Re: [ntdev] Paged or Non Paged ???

Thanks a bunch… That clarifies stuff.

From your stmt, Jake, now I assume that any driver that are loaded
much later in the system bring up ( meaning the Mm is active now )
phase are indeed not using pagefile !!!.

-pro

On Wed, Apr 29, 2009 at 4:24 PM, Jake Oshins
> wrote:
>
> The answer is simpler than you might expect. The memory manager can
> only page code from its original image if it knows where that image
> is. Boot drivers (which are the ones which are paged to the page
> file) are put into memory by the bootloader. Thus the memory manager
> has no idea where they came from on disk.
>
> –
> Jake Oshins
> Hyper-V I/O Architect
> Windows Kernel Group
>
> This post implies no warranties and confers no rights.
>
> --------------------------------------------------------------
>
> “Prokash Sinha” > wrote
> in message news:xxxxx@ntdev…
>
>
> Yes it is true that sys file goes to pagefile …
>
>
> Now the most non-obvious question is why they are jumbled together
> into pagefile ???
>
>
> If I’ve to guess, then (1) because they are relatively small in size
> (2) security (3) what else ???
>
>
> There are bits and pieces of informations here or there about this
> whole business of ( image format, in core process structure, who
> loads, paged vs. non-paged, privileges and all sorts )…
>
>
> In the past I’ve seen two articles on Mm & VA on insider - nicely
> written, could be nicer for newb ( and very very oldb, like me :).
> Something that would match with the general concepts one gets out of
> college ( and does not delve into it on a daily basis, but sometime
> need to get on with it ). This is exactly the
> poor-soul-from-the-driver-land needs !!!
>
>
> -pro
>
>
> On Wed, Apr 29, 2009 at 1:56 PM, Maxim S. Shatskih
> > wrote:
>
> Read something about NT file ( including image) mapping …
>
>
>
> User and kernel mode image mapping do differs a lot.
>
> User mapped image holds the file open, and pages in the pages from the
> image file itself on faults.
>
> Kernel driver does not hold the .SYS file open, and pages the pages in
> from the pagefile on faults.
>
> –
>
> Maxim S. Shatskih
> Windows DDK MVP
> xxxxx@storagecraft.com mailto:xxxxx
> http://www.storagecraft.com
>
>
> —
>
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
>
> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging
> and other seminars visit: http://www.osr.com/seminars To unsubscribe,
> visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx>

[Peter_Viscarola_OSR]

what are the other reasons that they have to be paged out ( if pageable sections presents) to pagefile

I know that at one time, drivers were always backed by the page file. This was implemented to allow the driver to be replaced while it was running, without having to stop the device.

Before this was implemented, the damn driver.sys file was locked until the driver was stopped. Made it a PITA for debugging… though I suspect it was actually implemented to allow drivers to take updates more easily.

Peter
OSR