Are userspace buffers segmented?

NTDEV
1

I have a driver that needs to reformat user-supplied data before sending
it on to the device. The data comprise one or more structures, each of
which may need to be internally padded, depending on content. The
userland buffer is allocated to allow plenty of room for the resulting
expansion of data.

I’m running into a IRQL_NOT_LESS_OR_EQUAL bugcheck when handling some
larger buffer sizes. User-side tests confirm successful allocation of
the buffer (around 18,700 bytes for this case). But the driver is
bugchecking inside of RtlMoveMemory. I’ve collected debugger data on a
bunch (10+) of crashes, and on every one of them the Arg1 ‘memory
referenced’ address is xxxxy000, where the destination address for
RtlMoveMemory was xxxx(y-1)fb0. It’s always crashing on a write and
never anywhere close to the end of the allocated buffer length.
(allocations have been consistent today, with a base address of
xxxxx250. yesterday, the base was xxxxx230 and one occurrance showed
xxxxy001 for Arg1)

Am I trying to write around the corner of a segment? (I didn’t think
P4s had segment issues) Do I need to avoid rewriting within the
user-supplied buffer?

Clues appreciated.

2

Is the user buffer virtually contigous?
Do you use embedded pointer?
What buffering method are you using?

Calvin Guan Software Engineer
ATI Technologies Inc. www.ati.com

-----Original Message-----
From: Roy Silvernail [mailto:xxxxx@parker.com]
Sent: Friday, September 03, 2004 12:00 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Are userspace buffers segmented?

I have a driver that needs to reformat user-supplied data
before sending
it on to the device. The data comprise one or more
structures, each of
which may need to be internally padded, depending on content. The
userland buffer is allocated to allow plenty of room for the
resulting
expansion of data.

I’m running into a IRQL_NOT_LESS_OR_EQUAL bugcheck when handling some
larger buffer sizes. User-side tests confirm successful
allocation of
the buffer (around 18,700 bytes for this case). But the driver is
bugchecking inside of RtlMoveMemory. I’ve collected debugger
data on a
bunch (10+) of crashes, and on every one of them the Arg1 ‘memory
referenced’ address is xxxxy000, where the destination address for
RtlMoveMemory was xxxx(y-1)fb0. It’s always crashing on a write and
never anywhere close to the end of the allocated buffer length.
(allocations have been consistent today, with a base address of
xxxxx250. yesterday, the base was xxxxx230 and one occurrance showed
xxxxy001 for Arg1)

Am I trying to write around the corner of a segment? (I didn’t think
P4s had segment issues) Do I need to avoid rewriting within the
user-supplied buffer?

Clues appreciated.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@ati.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

3

There shouldn’t be any segmentation issues.

From you mail it sounds like you’re modifying the user-buffer. Are you
trying to do this at raised IRQL? Are you properly probign the
user-mode buffer and wrapping all of your copies in try/except in case
the user-mode process invalidates the addresses?

I’m curious why you don’t require the user-mode process to put the data
in the correct format before trying to send it. It seems like you’re
adding complexity to a kernel mode component to make a user-mode one
simpler, which is really the wrong way around.

-p

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Roy Silvernail
Sent: Friday, September 03, 2004 9:00 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Are userspace buffers segmented?

I have a driver that needs to reformat user-supplied data
before sending it on to the device. The data comprise one or
more structures, each of which may need to be internally
padded, depending on content. The userland buffer is
allocated to allow plenty of room for the resulting expansion of data.

I’m running into a IRQL_NOT_LESS_OR_EQUAL bugcheck when
handling some larger buffer sizes. User-side tests confirm
successful allocation of the buffer (around 18,700 bytes for
this case). But the driver is bugchecking inside of
RtlMoveMemory. I’ve collected debugger data on a bunch (10+)
of crashes, and on every one of them the Arg1 ‘memory
referenced’ address is xxxxy000, where the destination
address for RtlMoveMemory was xxxx(y-1)fb0. It’s always
crashing on a write and never anywhere close to the end of
the allocated buffer length.
(allocations have been consistent today, with a base address
of xxxxx250. yesterday, the base was xxxxx230 and one
occurrance showed
xxxxy001 for Arg1)

Am I trying to write around the corner of a segment? (I
didn’t think P4s had segment issues) Do I need to avoid
rewriting within the user-supplied buffer?

Clues appreciated.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as:
xxxxx@windows.microsoft.com To unsubscribe send a blank
email to xxxxx@lists.osr.com

4

Peter Wieland wrote:

There shouldn’t be any segmentation issues.

From you mail it sounds like you’re modifying the user-buffer. Are you
trying to do this at raised IRQL?

Dispatch level, from an IOCTL call.

Are you properly probign the
user-mode buffer and wrapping all of your copies in try/except in case
the user-mode process invalidates the addresses?

I’m getting the user buffer with MmGetSystemAddressForMdl(). The user
process blocks until I’m done with the buffer, but the try/except is a
good point. (though would a try/except trap a bugcheck?)

I’m curious why you don’t require the user-mode process to put the data
in the correct format before trying to send it. It seems like you’re
adding complexity to a kernel mode component to make a user-mode one
simpler, which is really the wrong way around.

It’s a legacy support issue. The driver has to directly emulate a
driver for an older ISA device that didn’t require the internal padding.
I agree with you about the added complexity, but I’m not in management.

The curious point about this, of course, is that up to a certain size
transaction, everything works. Beyond that magic size, I get the
bugcheck when writing to xxxxy000.

5

> -----Original Message-----

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Roy Silvernail
Sent: Friday, September 03, 2004 10:28 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Are userspace buffers segmented?

Peter Wieland wrote:
> There shouldn’t be any segmentation issues.
>
> From you mail it sounds like you’re modifying the user-buffer. Are
> you trying to do this at raised IRQL?

Dispatch level, from an IOCTL call.

> Are you properly probign the
> user-mode buffer and wrapping all of your copies in
try/except in case
> the user-mode process invalidates the addresses?

I’m getting the user buffer with MmGetSystemAddressForMdl().
The user process blocks until I’m done with the buffer, but
the try/except is a good point. (though would a try/except
trap a bugcheck?)

Just because the user process you’re using blocks that doesn’t mean some
other evil process won’t try using the same interface to compromise the
system. Since you’re using a direct I/O control, you don’t need to
worry about the address space becoming invalid, so that’s not an issue
here.

However an evil process could modify the data it the buffer as you’re
moving it around. You need to be sure you’re validating all the data
that you use to drive the copying engine since, for example, the length
of a buffer could change from valid to invalid in the blink of an eye
(driving you to copy arbirrary and potentially sensitive kernel data
into their buffer.

And there’s the problem that direct I/O controls are supposed to be read
or write and shouldn’t really be modifying the input buffer. As long as
your clients aren’t expecting the data to remain untouched that may not
be a problem here though.

> I’m curious why you don’t require the user-mode process to put the
> data in the correct format before trying to send it. It seems like
> you’re adding complexity to a kernel mode component to make a
> user-mode one simpler, which is really the wrong way around.

It’s a legacy support issue. The driver has to directly
emulate a driver for an older ISA device that didn’t require
the internal padding.
I agree with you about the added complexity, but I’m not in
management.

The curious point about this, of course, is that up to a
certain size transaction, everything works. Beyond that
magic size, I get the bugcheck when writing to xxxxy000.

Yeah, that’s pretty odd. Is it possible that you’ve got some math
that’s screwing up at that magical boundary point?

-p

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as:
xxxxx@windows.microsoft.com To unsubscribe send a blank
email to xxxxx@lists.osr.com

6

But are you calling MmProbeAndLockPages to lock the entire 18K into memory
before attempting to copy using the pointer returned by
MmGetSystemAddressForMdl (oh and don’t forget to use the Safe() variant of
course!)?

You need the try/except around the MmProbaAndLockPages to handle the case
where the usermode code gives you a garbage buffer address.

The failure is happening on a pageboundary so perhaps you are just hitting a
page that is currently invalid (i.e. paged out).

/simgr

-----Original Message-----
From: Roy Silvernail [mailto:xxxxx@parker.com]
Sent: Friday, September 03, 2004 1:28 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Are userspace buffers segmented?

Peter Wieland wrote:

There shouldn’t be any segmentation issues.

From you mail it sounds like you’re modifying the user-buffer. Are you
trying to do this at raised IRQL?

Dispatch level, from an IOCTL call.

Are you properly probign the
user-mode buffer and wrapping all of your copies in try/except in case
the user-mode process invalidates the addresses?

I’m getting the user buffer with MmGetSystemAddressForMdl(). The user
process blocks until I’m done with the buffer, but the try/except is a
good point. (though would a try/except trap a bugcheck?)

I’m curious why you don’t require the user-mode process to put the data
in the correct format before trying to send it. It seems like you’re
adding complexity to a kernel mode component to make a user-mode one
simpler, which is really the wrong way around.

It’s a legacy support issue. The driver has to directly emulate a
driver for an older ISA device that didn’t require the internal padding.
I agree with you about the added complexity, but I’m not in management.

The curious point about this, of course, is that up to a certain size
transaction, everything works. Beyond that magic size, I get the
bugcheck when writing to xxxxy000.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@stratus.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

7

I was under the impression that he is using direct IO and already has an
MDL, so no MmProbeAndLockPages is required. I think he means “getting access
to the user data” rather than literally Irp->UserBuffer.

Perhaps he ought to be more specific in describing the code that actually
causes the crash.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Graham, Simon
Sent: Friday, September 03, 2004 2:24 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Are userspace buffers segmented?

But are you calling MmProbeAndLockPages to lock the entire
18K into memory before attempting to copy using the pointer
returned by MmGetSystemAddressForMdl (oh and don’t forget to
use the Safe() variant of course!)?

You need the try/except around the MmProbaAndLockPages to
handle the case where the usermode code gives you a garbage
buffer address.

The failure is happening on a pageboundary so perhaps you are
just hitting a page that is currently invalid (i.e. paged out).

/simgr

-----Original Message-----
From: Roy Silvernail [mailto:xxxxx@parker.com]
Sent: Friday, September 03, 2004 1:28 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Are userspace buffers segmented?

Peter Wieland wrote:
> There shouldn’t be any segmentation issues.
>
> From you mail it sounds like you’re modifying the user-buffer. Are
> you trying to do this at raised IRQL?

Dispatch level, from an IOCTL call.

> Are you properly probign the
> user-mode buffer and wrapping all of your copies in
try/except in case
> the user-mode process invalidates the addresses?

I’m getting the user buffer with MmGetSystemAddressForMdl().
The user process blocks until I’m done with the buffer, but
the try/except is a good point. (though would a try/except
trap a bugcheck?)

> I’m curious why you don’t require the user-mode process to put the
> data in the correct format before trying to send it. It seems like
> you’re adding complexity to a kernel mode component to make a
> user-mode one simpler, which is really the wrong way around.

It’s a legacy support issue. The driver has to directly
emulate a driver for an older ISA device that didn’t require
the internal padding.
I agree with you about the added complexity, but I’m not in
management.

The curious point about this, of course, is that up to a
certain size transaction, everything works. Beyond that
magic size, I get the bugcheck when writing to xxxxy000.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as:
xxxxx@stratus.com To unsubscribe send a blank email to
xxxxx@lists.osr.com

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as:
xxxxx@hollistech.com To unsubscribe send a blank email to
xxxxx@lists.osr.com

8

Agreed, Mark.

Just to review the checklist:

  • The DeviceIoControl code definition must use METHOD_OUT_DIRECT.
    METHOD_IN_DIRECT only checks the buffer to see that the pages are
    readable, and this is not sufficient here.

(Mnemonic: The “IN” or “OUT” in these codes is in the sense of the driver
looking at the app. So METHOD_OUT_DIRECT is for data being output from
the driver to the app.)

  • The app passes the buffer in question via the lpOutBuffer and
    nOutBufferSize args to DeviceIoControl.

  • The driver checks for a valid device IO control code, checks for a valid
    (according to your interface specs) buffer length in
    IrpStackLocation->Parameters.DeviceIoControl.OutputBufferLength, and then
    calls

system_Space_address_for_users_buffer =
MmGetSystemAddressForMdlSafe(Irp->MdlAddress,NormalPriority);

Do not use MmProbeAndLockPages, try-except, etc.

  • after checking the return value here for NULL, the driver can then use
    the returned value as a pointer to the user’s buffer, for both reading and
    writing. There will be no page faults, no segmentation issues, etc. This
    may be done at any time, at any IRQL, as long as the IRP has not yet been
    passed to IoCompleteRequest. Ignore the UserBuffer field of the IRP.

  • If the buffer has an internal structure that the driver must “walk,”
    with fields that give or imply the length of substructures, then the
    driver has to be careful that the total “internal” length so implied never
    exceeds the original length. The driver must never access any bytes
    beyond

system_Space_address_for_users_buffer +
IrpStackLocation->Parameters.DeviceIoControl.OutputBufferLength - 1

  • If the app may be modifying the contents of the buffer after sending the
    IRP to the driver, it is reasonable to copy the buffer somewhere else and
    then deal with the copy. If the internal structure of the copy is valid,
    fine; if not, complete the IRP with error status.

  • Once again, do not dereference the pointer returned by
    MmGetSystemAddressForMdlSafe after you have passed the IRP to
    IoCompleteRequest. (Yes, nitpickers, there are exceptions. They all fall
    under the category of “bad coding practice,” except for the ones that
    cannot *even* be considered bad.)

If all of these points are followed I just don’t see how the error being
reported by the OP is possible.

> transaction, everything works. Beyond that magic size, I get the
> bugcheck when writing to xxxxy000.

Exactly what address would that be?

What does the debugger’s !pte command tell you about that address?

Have you looked at the buffer pointer, etc., with the debugger? !pte
command on the address, look at the IRP and the MDL, etc.?

— Jamie Hanrahan
Kernel Mode Systems http://www.cmkrnl.com/
Azius Developer Training http://www.azius.com/
Windows internals and drivers - consulting, training, and debugging

9

(back from the vacation interval)

Jamie Hanrahan wrote:

Agreed, Mark.

Just to review the checklist:

  • The DeviceIoControl code definition must use METHOD_OUT_DIRECT.
    METHOD_IN_DIRECT only checks the buffer to see that the pages are
    readable, and this is not sufficient here.

We have a winner! The IOCTL specifies METHOD_IN_DIRECT. The driver is
attempting to write to that buffer, and when it crosses a page boundary,
it bugchecks. Gotta love legacy code! (the write business was added to
the PCI version of this driver so that it could slide into an existing
ISA installation seamlessly… looks like they missed a seam)

Thanks for an excellent checklist, Jamie. I’m putting a copy in my
notebook.