When do BUFFERED and NEITHER methods break even?

Don, this whole discussion got me completely confused and continues to confuse me more and more. For years I’ve been pretty sure that all I need to do for “raw”, self-contained (i.e. no embedded pointers / offsets pointing out of the buffer) user buffers, passed in NEITHER IOCTL or in Fast I/O path (device I/O, read/write, etc.), all I needed was to MDL that raw buffer, lock it (with proper access method / requestor mode), get system address and I’m good to do with it whatever I’m pleased, respecting access method. Your initial comment made me think that I’m missing some important checks that I’m just unaware of. And I was really wandering what these checks are. The way I read your last comment (about reducing NEITHER to DIRECT) suggests, that I was right in my assumptions about how to handle raw, self-contained user buffers and MDL + proper lock is all what IOMgr does for DIRECT. Don’t get me wrong, I’m not trying to poor more gas into the flame, I just want to clarify my childhood beliefs. Were they wrong, or not? And if they were wrong, I’d really wanna know, where I screwed up, what checks I missed, and all that jazz.

I guess *I’M* the one that’s confused.

Why in the world would you use METHOD_NEITHER, and then create an MDL, probe and lock the buffer, and get a system address? As I think Don already said, you’ve just written the code for METHOD_xxx_DIRECT yourself, which seems to completely eliminate any advantage to using METHOD_NEITHER in the first place.

Note that even in this case (as with METHOD_xxx_DIRECT) you have the security issue of the user having simultaneous access to the data buffer while you’re accessing it in your driver. Not a problem, as long as the buffer doesn’t contain any lengths or pointers, or if you capture and then validate those properly.

But, back to the case at hand. You’ve asked what you need to do to properly use METHOD_NEITHER. I’ll tell you…

First, you’ll need to be sure you check and handle the case of zero-length buffers. You cannot rely on ProbeForRead and ProbeForWrite to catch these for you… a zero length buffer is a valid buffer, as any reader of NTFSD should already know.

METHOD_NEITHER has the singular distinction of having NO (none, zero) validation done on either the InBuffer or OutBuffer pointers or lengths. This means that the caller can supply arbitrary 32-bit values for these parameters, and the I/O Manager will pass those arbitrary parameters to your driver without any validation or review.

THUS, it’s entirely the job of your driver to validate these parameters. If the parameters are supposed to describe a buffer in user address space, then you must CORRECTLY use ProbeForRead or ProbeForWrite on the buffer. All access to the buffer must be within an exception handler. Be sure to clean-up correctly from any exceptions.

If the buffer you’re validating contains pointers, you must capture the buffer (that is, make a separate copy of the buffer in kernel virtual address space) after validating the buffer pointers. THEN, validate any contained pointers (again, in an exception handler) using ProbeForRead or ProbeForWrite. “Rinse and repeat” for any other pointers.

Again: ENSURE all access to the buffer is done in a try/except block.

That’s really “all” there is to it. Do it right, not a problem. Get it wrong, blue screen the system. It’s not magic, it’s just annoying.

Of course, the above assumes that the data buffer must reside in user space. If the data buffer CAN reside in Kernel space, note that ProbeForRead and ProbeForWrite will flag a kernel-mode buffer as invalid.

It’s not a mystery… it’s not that exotic… it’s not an urban legend. It’s just plain annoying, gritty, detailed code to write and get correct. And it’s not for noobs and wannabes.

Noobs see Neither I/O (or METHOD_NEITHER) and they get all excited. They often think it sounds like a great way to reduce overhead in their driver, make it really fast, and lets them do something different. This happens particularly when they hit the stage where they start to understand how context is handled in Windows, and how the first driver entered by the I/O Manager is called in the context of the requesting thread. So, they’re “experts” (not) and they want to play with something neat. This is the stuff of which bad drivers are made.

In the many years I’ve been writing Windows drivers, I’ve seen a few examples of places where METHOD_NEITHER is either (a) Required, or (b) Highly advisable. But there are darn few. And most of the cases I find it’s use, I usually find a bug.

Trust me: We just finished fixing MORE THAN 100 separate bugs in a driver written by somebody who “knew” how to use METHOD_NEITHER IOCTLs. A real genius…

Peter
OSR

Peter: “Why in the world would you use METHOD_NEITHER, and then create an MDL, probe and
lock the buffer, and get a system address?”
*
'Cause this is the only *generic, system provided* way to handle IOCTLs that *by “ideal design”* require both, in and out buffers being (possibly) large. And again, *by “ideal design”* “large” means large enough to compensate MDL’ing overhead and make NEITHER more performant. And this “ideal large” is about 2 pages big. Let it be even 4 pages big. Which is not big at all if you consider “inverted call” machinery (an example that I gave in this thread) that transport large kernel requests and large user responses. In my case majority of those both-ways buffers reach up to 64K and will greatly exceed this size on Vista & up.
*
Another thing is that if you try to squieeze some more “precious ticks” utilizing IRP-less “Fast I/O way” (again, for the sake of that “ideal design”), you also end up with raw buffers, regardless of the buffering method that’s being used.
*
Now, coming back from this “ideal design” thing to the “real world”, I’m agreed with all your points that they must be taken and cosider seriously. And I indeed consider some of the possible NEITHER Fast I/O paths not worth the effort presisely because of one of the points that you made: it’s impossible (well, at least too hard for me to do it right) to distinguishing between kernel and user buffers in this path which makes it highly vulnerable for screwing things up. Of course, with all these problems one may prefer just to refuse using NEITHER and Fast I/O all together and find a substitute in combining DIRECTs (which by itself complicates design, not just implementation!, and as such introes possibilities for even more and worst errors). But at least do it with clear understanding of why you do this. And likewise, know the dangers and design the system to minimize that danger. For me the rules are: "no inside-out pointers, all the “internal offsets” are copied out before analyzing / validating them, no kernel requestors are accepeted on Fast I/O / NEITHER path.

Whatever. I’m sure you have your reasons. You asked, I gave you an answer. Do as you like.

You would think that, wouldn’t you. Most intelligent people would. But would you believe that I’ve seen cases where ELIMINATING FAST I/O actually increased the performance of the underlying file system by something like 20%?

No kidding. Remember the wise words of OSR Consulting Partner Tony Mason: “Fast I/O… it’s neither fast, nor does it necessarily have anything to do with I/O.”

But, now we’re way, way, off the original thread’s purpose…

Peter
OSR

“But, now we’re way, way, off the original thread’s purpose”
*
Actually, we were off it from the very beginning :slight_smile: Probably, this “N” word is just as much cursed, as these two notorious “addition” signs. I will be more cautious next time.
*
But in any case, thanks a lot for taking your time and giving constructive answers. It’s always a pleasure to hear from you. And although you intrigued me with your last comment about 20%, I’ll keep this Q for Tony to answer… May be at the next plugfest. In any case, I’ve learned something, so thanks a lot!

Cheers

For whatever it is worth, I commend your study of the situation. I was
going to post this right after I read it, but I knew that this one was
going to get out of hand. That I agree with a some of Don’s and Peter’s
points notwithstanding, I hear you on discovery for its own sake, and
it’s nice to see someone take the time to do it and publish it. I
certainly agree with your statement regarding overreaction/overstatement
of difficulty/threat, and am guilty of it sometimes myself, and that is
what I expected to happen here, although I would say that METHOD_NEITHER
does tend to always be a bit of a disaster, which is why I passed
originally.

In any case, it was interesting.

Incidentally, You lost me on the “addition” signs???

Thanks,

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@gmail.com
Sent: Wednesday, September 05, 2007 18:33
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] When do BUFFERED and NEITHER methods break even?

“But, now we’re way, way, off the original thread’s purpose”
*
Actually, we were off it from the very beginning :slight_smile: Probably, this “N”
word is just as much cursed, as these two notorious “addition” signs. I
will be more cautious next time.
*
But in any case, thanks a lot for taking your time and giving
constructive answers. It’s always a pleasure to hear from you. And
although you intrigued me with your last comment about 20%, I’ll keep
this Q for Tony to answer… May be at the next plugfest. In any case,
I’ve learned something, so thanks a lot!

Cheers


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@evitechnology.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

“Incidentally, You lost me on the “addition” signs???”
*
Lucky you! :wink:
No, really, you’re lucky. It’s one of the cases when ignorance is really a blessing. It’s just that this is also one of the condemned words in drv dev area :slight_smile: My bad that I didn’t follow this forum for a long time and didn’t know that “n” word is also condemned :slight_smile:
*
Sorry for that digression, but this thread is finished anyway.

Didn’t you forget to mention that “in addition, the driver must copy the
data to a safe kernel-mode address in the pool or on the stack before
manipulating it. Copying the data to a kernel-mode buffer ensures that the
user-mode caller cannot change the data after the driver has validated it.”
? (from “what every driver write needs to know”)

/Daniel

wrote in message news:xxxxx@ntfsd…
> That’s really “all” there is to it.

I mean you mentioned that but only in the case of embedded pointers. You
should always copy both the input and output buffers, shouldn’t you ?

/Daniel

“Daniel Terhell” wrote in message
news:xxxxx@ntfsd…
> Didn’t you forget to mention that “in addition, the driver must copy the
> data to a safe kernel-mode address in the pool or on the stack before
> manipulating it. Copying the data to a kernel-mode buffer ensures that the
> user-mode caller cannot change the data after the driver has validated
> it.” ? (from “what every driver write needs to know”)
>
> /Daniel
>
>
>
> wrote in message news:xxxxx@ntfsd…
>> That’s really “all” there is to it.
>
>

There’s usually no need to do so (see more below). If there was, there’d be never be any advantage to METHOD_NEITHER over METHOD_BUFFERED because they’d always be 100% equivalent.

If the data buffer contains data that your driver must somehow interpret and/or validate, then you need to prevent that data changing over time. What you’re trying to avoid is the case where:

(T1) Your driver examines and validates the data passed in the buffer
(T2) Some other user thread modifies the data in a bad way
(T3) Your driver uses the data in the buffer, believing it is valid… when it in fact has been altered since it was checked.

To avoid this, the driver first “captures” the data by making a private copy. That COPY is then validated, and subsequent operations are carried out on the validated COPY.

Note that we’re NOT at all concerned with the case where the requestor sends a buffer containing pure data for the driver to write and then modifies that data before, during, or after the write operation. While this may result in “bad” or corrupted data being written by the driver, it’s an error in the requestor’s logic and no concern of the driver’s whatsoever.

To be comprehensive: Note that we also shouldn’t typically be concerned with data coming from other kernel-mode requestors. By the standard rules of Windows architecture, kernel-mode components implicitly trust other kernel-mode components. So there’s no need to “capture” data from kernel-mode requestors. The idea here is that if a kernel-mode requestor wanted to do something malicious, they could do it a lot easier than passing an invalid data buffer.

Peter
OSR

Hi, and sorry for the somewhat late post.

From my experiments, one can reach more than 1000 calls per ms.

So I guess that I was testing on an X64 computer that is probably
stronger than the computer that you are using, but there are some things
that one might try and do to have things work faster.

First thing is open the device with flag FILE_FLAG_OVERLAPPED (probably
trivial).

Second thing is use fast IO. This means no IRP should be created. This
is a documented way although very hard to find the documentation.

The third thing is use DeviceIoControl() with NULL in the overlapped
parameter. MSDN says this won’t work. From my experiments, this indeed
doesn’t work unless you remember one rule. The driver must return
success to the caller in the user mode. (this means the FastDispatch
must always return true and IRP is never allowed to be created). If you
want to return an error or even pending, find another way to do it (for
example use the buffer that you are allowed to return).

All that was said about validating the buffers is of course correct and
has to be done.

And for the obvious question, is the extra performance worth the extra
work?
Well we create drivers for 20GB cards. In order to reach full wire speed
with 1kb buffers, we must be able to move from user to kernel and back
in less than a microsecond.

Thanks
Tzachi

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@gmail.com
Sent: Wednesday, September 05, 2007 5:58 PM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] When do BUFFERED and NEITHER methods break even?

So, I ran some tests :wink: and got puzzling (well, puzzling to
me) results. All tests were run as such:
For each pair (method, buffer_size) in a time-restricted loop
(4 seconds for each pair) I was issuing (and counting number
of) synchronous DeviceIoControl calls to my test driver (no
verifier attached). For each IOCTL both, in and out buffers
had same size (buffer_size) and both were page-aligned. In
the driver, for NEITHER method I create MDL, lock it and get
mapped system address. After that (and in BUFFERED case
starting with this) I just complete IRP with
IO_STATUS.Information set to IOCTL’s input buffer length. So,
there is no data copy is involved, just pure minimal overhead
calculation. Result was sort of expected with one puzzling
exception: a sudden (and significant) drop in BUFFERED method
performance around 1.5K - 3.5K buffer size region. This drop
is pretty much consistent. No matter how many times I run the
test, it’s always there at least for one buffer size from
this range. The machine I ran this test is HT single CPU with
3G of memory. Probably, I should Kernrate that test to get an
idea where is the source for the drop, but before starting
that extra work, I’d rather ask the experts :wink: So, does
anybody have an idea why I see what I see?
Here are the stats that I collected:

Method: BUFFERED, Buffer size = 6144, Requests 336257, Rate =
84 req / msec
Method: NEITHER, Buffer size = 6144, Requests 387828, Rate =
96 req / msec

Method: BUFFERED, Buffer size = 5632, Requests 341320, Rate =
85 req / msec
Method: NEITHER, Buffer size = 5632, Requests 387961, Rate =
96 req / msec

Method: BUFFERED, Buffer size = 5120, Requests 341749, Rate =
85 req / msec
Method: NEITHER, Buffer size = 5120, Requests 387452, Rate =
96 req / msec

Method: BUFFERED, Buffer size = 4608, Requests 345939, Rate =
86 req / msec
Method: NEITHER, Buffer size = 4608, Requests 389834, Rate =
97 req / msec

Method: BUFFERED, Buffer size = 4096, Requests 350218, Rate =
87 req / msec
Method: NEITHER, Buffer size = 4096, Requests 402132, Rate =
100 req / msec

Method: BUFFERED, Buffer size = 3584, Requests 380733, Rate =
95 req / msec
Method: NEITHER, Buffer size = 3584, Requests 400632, Rate =
100 req / msec

Method: BUFFERED, Buffer size = 3072, Requests 110274, Rate =
27 req / msec
Method: NEITHER, Buffer size = 3072, Requests 400716, Rate =
100 req / msec

Method: BUFFERED, Buffer size = 2560, Requests 122674, Rate =
30 req / msec
Method: NEITHER, Buffer size = 2560, Requests 402039, Rate =
100 req / msec

Method: BUFFERED, Buffer size = 2048, Requests 123675, Rate =
30 req / msec
Method: NEITHER, Buffer size = 2048, Requests 400389, Rate =
100 req / msec

Method: BUFFERED, Buffer size = 1536, Requests 420653, Rate =
105 req / msec
Method: NEITHER, Buffer size = 1536, Requests 402658, Rate =
100 req / msec

Method: BUFFERED, Buffer size = 1024, Requests 431754, Rate =
107 req / msec
Method: NEITHER, Buffer size = 1024, Requests 402271, Rate =
100 req / msec

Method: BUFFERED, Buffer size = 512, Requests 457249, Rate =
114 req / msec
Method: NEITHER, Buffer size = 512, Requests 402445, Rate =
100 req / msec


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: unknown lmsubst tag
argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com