Re: MmGetMdlByteCount greater than TransferBufferLength

You know, I’ve started to answer this thread three separate times, and each time I’ve lost the will to argue and deleted my answer.

Mr. Sykes… with all due respect… from a Windows architectural perspective, you’re just wrong. Take this as an opportunity to learn something, instead of as an opportunity to argue. Mr. Roberts is rarely incorrect about such fundamental concepts, and he’s absolutely correct here.

I will admit: If you spend most of your time in the upper parts of the storage stack (for example), it’s pretty easy to see the MDL as “the definitive description of the transfer”… but, really, it’s not.

It turns out that there are *some* places were the MDL size and the I/O operation (read/write) length are identical. But any time there is a separate place to get the length of the operation (URB, I/O Stack, SRB) it is incorrect to rely on the size of the underlying MDL as the size of the operation.

As Mr. Roberts and Mr. Grig described, the MDL describes the size of a buffer in kernel mode. That’s all it represents. When one performs an I/O operation from the described buffer, there is not ALWAYS the implication that the MDL describes the I/O operation.

Whenever there’s a description of the I/O operation… pointer, length, whatever… that’s separate from the MDL, you use the separate description. The MDL has to be valid. And the actual data buffer used for the transfer has to be a valid subset of the buffer described by the MDL. That subset can be 100% of the MDL or just part of the MDL.

This is just the way Windows architecture has been, since… well… at least 1992 when I started working with Windows NT.

Peter
OSR
@OSRDrivers

Two thumbs up to Peter!

MDL describe the buffer, not the transfer. The concept is not so difficult
to understand. There can be many, many reasons why the buffer is larger;
alignment, separate transfers that break a large IO into smaller IOs, &c.

On Sat, Feb 4, 2017 at 10:57 AM wrote:

>


>
> You know, I’ve started to answer this thread three separate times, and
> each time I’ve lost the will to argue and deleted my answer.
>
> Mr. Sykes… with all due respect… from a Windows architectural
> perspective, you’re just wrong. Take this as an opportunity to learn
> something, instead of as an opportunity to argue. Mr. Roberts is rarely
> incorrect about such fundamental concepts, and he’s absolutely correct here.
>
> I will admit: If you spend most of your time in the upper parts of the
> storage stack (for example), it’s pretty easy to see the MDL as “the
> definitive description of the transfer”… but, really, it’s not.
>
> It turns out that there are some places were the MDL size and the I/O
> operation (read/write) length are identical. But any time there is a
> separate place to get the length of the operation (URB, I/O Stack, SRB) it
> is incorrect to rely on the size of the underlying MDL as the size of the
> operation.
>
> As Mr. Roberts and Mr. Grig described, the MDL describes the size of a
> buffer in kernel mode. That’s all it represents. When one performs an I/O
> operation from the described buffer, there is not ALWAYS the implication
> that the MDL describes the I/O operation.
>
> Whenever there’s a description of the I/O operation… pointer, length,
> whatever… that’s separate from the MDL, you use the separate
> description. The MDL has to be valid. And the actual data buffer used for
> the transfer has to be a valid subset of the buffer described by the MDL.
> That subset can be 100% of the MDL or just part of the MDL.
>
> This is just the way Windows architecture has been, since… well… at
> least 1992 when I started working with Windows NT.
>
> Peter
> OSR
> @OSRDrivers
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: <
> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

Further to Peter?s answer (and I too have essayed to enter heretofore and also have deleted some nascent works)

You may have a legitimate issue with the document writers at Microsoft for not making this point more clear by using identical language for two different values, but the main problem is this is obvious. First, from the point of view that when approaching any API (DDI) the presence of two separate parameters indicates that they describe two logically different values (independent or otherwise) and second from the point of view that at a systemic level how would it be possible to implement in a different way.

You may justly criticise the doc writers for not including the correct references to other concepts or failing to qualify the count of which bytes, but when in this history of computing has any documentation ever been definitive from the point of view of the arbitrarily uninured; or for that matter in the history of writing, as each document must perforce assume something from its audience including at least the literacy to read its words.

I have often found that it is important to think before blindly blaming others for their miss-work on the design or documentation of hardware or software. This method has stood me well through every kind of technology I have ever encountered except for cryptography. In this area Microsoft assuredly has some of their worst APIs and even poorer documentation, but equally assuredly they are well advanced compared with others who participate.

Sent from Mailhttps: for Windows 10

From: xxxxx@osr.commailto:xxxxx
Sent: February 4, 2017 10:57 AM
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: RE:[ntdev] Re: MmGetMdlByteCount greater than TransferBufferLength



You know, I’ve started to answer this thread three separate times, and each time I’ve lost the will to argue and deleted my answer.

Mr. Sykes… with all due respect… from a Windows architectural perspective, you’re just wrong. Take this as an opportunity to learn something, instead of as an opportunity to argue. Mr. Roberts is rarely incorrect about such fundamental concepts, and he’s absolutely correct here.

I will admit: If you spend most of your time in the upper parts of the storage stack (for example), it’s pretty easy to see the MDL as “the definitive description of the transfer”… but, really, it’s not.

It turns out that there are some places were the MDL size and the I/O operation (read/write) length are identical. But any time there is a separate place to get the length of the operation (URB, I/O Stack, SRB) it is incorrect to rely on the size of the underlying MDL as the size of the operation.

As Mr. Roberts and Mr. Grig described, the MDL describes the size of a buffer in kernel mode. That’s all it represents. When one performs an I/O operation from the described buffer, there is not ALWAYS the implication that the MDL describes the I/O operation.

Whenever there’s a description of the I/O operation… pointer, length, whatever… that’s separate from the MDL, you use the separate description. The MDL has to be valid. And the actual data buffer used for the transfer has to be a valid subset of the buffer described by the MDL. That subset can be 100% of the MDL or just part of the MDL.

This is just the way Windows architecture has been, since… well… at least 1992 when I started working with Windows NT.

Peter
OSR
@OSRDrivers


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></mailto:xxxxx></mailto:xxxxx></https:>

The replies to this topic have been excellent. The mindset of the OP is incompatible with kernel development and surely makes many hope his drivers never run on their systems. Yes it is true the kernel APIs are convoluted, error prone, and documentation often problematic. You have to put processes together to deal with this to be successful. Over the years the most reliable thing I have found to do when there is ever a doubt is defer to what the samples do. Since they were written by knowledgeable people, get executed and tested and updated, they do things the correct way. So a simple peek at a relevant source file was all that was necessary to confirm any doubt in cases such as this.

> This is just the way Windows architecture has been, since… well… at least 1992

when I started working with Windows NT.

Oh dear…

You seem to have had started working with NT before it even got released…

https://en.wikipedia.org/wiki/Windows_NT

To be honest, I had a very remote idea of what the PC looked like at the time anyway…

Anton Bassov

xxxxx@hotmail.com wrote:

> This is just the way Windows architecture has been, since… well… at least 1992
> when I started working with Windows NT.
Oh dear…

You seem to have had started working with NT before it even got released…

Many people did. The development of what was originally called NT OS/2
began in 1988 or 1989.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

@Everyone, I find it staggering that the very simple statement of mine that:

  1. Either the documentation needs improving to describe theses values better, or
  2. The driver above has put a wrong value in one of them

Is not accepted as a perfectly valid statement, since it clearly is factual.

And now with the clarification from Tim and Anton we know its the documentaion that is insufficient.

@Sven. Trusting the sample code is a very bad idea. It is good for ideas, but has always been very buggy. To use your suggestion for example, take a look at this line from the sample code:

length = min(MmGetMdlByteCount(irp->MdlAddress), pMpRfd->PacketSize);
RtlCopyMemory(buffer, pMpRfd->Buffer, length);

from the 7600 DDK, in the pcidrv\wdm\hw source. It is using the byte count as the copy length, not the irp value.

Ok so how can it be that the size of a request is (much) greater than the size of MDL associated with it? Do newer Windows versions (where the ioctl buffer size is not limited by max. MDL size) create something like array of MDLs to pass a large buffer? Then one such request can span several MDLs?

Regards,
– pa

@Pavel, if there are a chain of MDLs does their total byte count look something like the transfer buffer size?

xxxxx@hotmail.com wrote:

@Everyone, I find it staggering that the very simple statement of mine that:

  1. Either the documentation needs improving to describe theses values better, or
  2. The driver above has put a wrong value in one of them

Is not accepted as a perfectly valid statement, since it clearly is factual.

And now with the clarification from Tim and Anton we know its the documentaion that is insufficient.

Which documentation? I don’t claim that the documentation is perfect,
but I’d like to know which documents set you astray.

@Sven. Trusting the sample code is a very bad idea. It is good for ideas, but has always been very buggy. To use your suggestion for example, take a look at this line from the sample code:

length = min(MmGetMdlByteCount(irp->MdlAddress), pMpRfd->PacketSize);
RtlCopyMemory(buffer, pMpRfd->Buffer, length);

from the 7600 DDK, in the pcidrv\wdm\hw source. It is using the byte count as the copy length, not the irp value.

This is a completely different situation. You were specifically talking
about USB URBs, where requests can come from arbitrary kernel-mode
drivers. Kernel-mode programmers have the flexibility to do magical
things with their MDLs, so the lengths don’t need to match. In that
case, the length of the transfer is given in the URB field.

In the pcidrv example, this is known to be an IRP_MJ_READ, and the
contract is that it is coming from a user-mode client using direct I/O.
In that case, we KNOW that the MDL exactly describes the user’s buffer,
because the operating system I/O manager always creates them that way.
This design alleviates the need to carry the length as a separate piece
of state.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

>In that case, we KNOW that the MDL exactly describes the user’s buffer,
because the operating system I/O manager always creates them that way.
This design alleviates the need to carry the length as a separate piece
of state.

Actually, IO_STACK_LOCATION::Read carries Length, as well (Write does that, too), which takes priority over whatever is in MDL. The MDL will most likely have the same length, though.

@Tim The MSDN documentation on the MDL buffer length, and the URB transfer buffer length.

https://msdn.microsoft.com/en-us/library/windows/hardware/ff554530(v=vs.85).aspx "MmGetMdlByteCount returns the length, in bytes, of the buffer described by Mdl. "

https://msdn.microsoft.com/en-us/library/windows/hardware/ff540352(v=vs.85).aspx "TransferBufferLength

Specifies the length, in bytes, of the buffer specified in TransferBuffer or described in TransferBufferMDL"

See what I mean? They are documented as being one and the same. If it is legal for a driver to act as you and Anton say then the documentation should be changed since as it stands it doesnt permit it.

Your comments regarding my example merely compound the lack of documentation. You do resalise what you are saying here Tim, that it depends on who created the MDL-Irp-URB as to whether the two byte count values are the same or not?

Really? And the code is supposed to know this is it? How about if an upper filter is put between between your driver that trusts the IoManager, and suddenly your driver hasa byte count thats wrong?

BSOD.

Come on Tim, this has to be tightened up. I say it again, either the documentation is wrong or the driver that preps an Irp with different byte count values is wrong.

Trying to have it both ways is chaos, utter chaos.

@Peter, and can I add I am NOT saying Tim is wrong in his description of the way the MDL byte count is handled, I am saying if he is right, then he is wrong to state the documentation describes such a handling. Because it doesn’t.

I don’t know how you misread my replies to Tim, I thought my popsition was, and is, quite clear from them.

You keep confusing two things. An MDL and an I/O request.
An MDL just describes some memory.
You do I/O with whatever the I/O request says. If it has Length and Address fields, you use these fields. If it has an MDL, you pass the MDL to GetScatterGatherList, but still use URB/SRB length and address in GetScatterGatherList call.

And it keeps going…

On Wed, Feb 8, 2017 at 9:35 AM wrote:

> You keep confusing two things. An MDL and an I/O request.
> An MDL just describes some memory.
> You do I/O with whatever the I/O request says. If it has Length and
> Address fields, you use these fields. If it has an MDL, you pass the MDL to
> GetScatterGatherList, but still use URB/SRB length and address in
> GetScatterGatherList call.
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: <
> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

xxxxx@hotmail.com wrote:

@Tim The MSDN documentation on the MDL buffer length, and the URB transfer buffer length.

https://msdn.microsoft.com/en-us/library/windows/hardware/ff554530(v=vs.85).aspx "MmGetMdlByteCount returns the length, in bytes, of the buffer described by Mdl. "

That’s true.

https://msdn.microsoft.com/en-us/library/windows/hardware/ff540352(v=vs.85).aspx “TransferBufferLength
specifies the length, in bytes, of the buffer specified in TransferBuffer or described in TransferBufferMDL”

I will grant you that this could be worded more precisely. It should
say that TransferBufferLength specifies the length, in bytes, of the
buffer to be used for this transfer.

I assume you can understand the difficulty here. I can allocate a large
buffer, and then use parts of that buffer for different purposes. When
you get a TransferBuffer in an URB, the length is not carried along, so
there is no conflict. When you get a TransferBufferMDL, it just so
happens that the MDL carries knowledge of its own length, but that
length is not relevant to this transfer.

Your comments regarding my example merely compound the lack of documentation. You do resalise what you are saying here Tim, that it depends on who created the MDL-Irp-URB as to whether the two byte count values are the same or not?

Right. It absolutely does. Different driver stacks have different
contracts, largely because of history. That’s just the way it is.

Really? And the code is supposed to know this is it? How about if an upper filter is put between between your driver that trusts the IoManager, and suddenly your driver hasa byte count thats wrong?

BSOD.

Correct. That’s what happens when a driver violates the contract, which
is what such a filter driver would have done. An upper-level driver is
ALLOWED to assume conditions from I/O Manager. Any upper filter must
maintain those conditions.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

In fact they even had a partner program for application software. The highlight of which for me was getting a fancy paperweight with the OS/2 logo suspended in a triangular prism of plastic

Sent from Mailhttps: for Windows 10

From: Tim Robertsmailto:xxxxx
Sent: February 6, 2017 2:32 PM
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: Re: [ntdev] Re: MmGetMdlByteCount greater than TransferBufferLength

xxxxx@hotmail.com wrote:
> > This is just the way Windows architecture has been, since… well… at least 1992
>> when I started working with Windows NT.
> Oh dear…
>
> You seem to have had started working with NT before it even got released…

Many people did. The development of what was originally called NT OS/2
began in 1988 or 1989.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></mailto:xxxxx></mailto:xxxxx></https:>

>Many people did. The development of what was originally called NT OS/2 began in 1988 or 1989

Well, normally I would expect them to be mainly on the development team, but, OTOH, any OS
needs some hardware to run on, and,hence, some third-party drivers may be included in the very first OS release. Therefore, a bit of OS work may be,indeed, done by the third-party developers even prior to its first official release…

Anton Bassov

Anton

For OS/2, they had programs for all kinds of ISVs. They wanted to have applications that actually ran on the new OS when it launched , so they engaged a wide audience. At the time I was working on Line of Business applications for the financial industry (as I am again) rather than drivers and we were assigned a project to build an OS/2 version of our software. Our life became radially simpler once Microsoft decided to implement the Win32 subsystem (or was it called Win16 of something I forget; or I?m just getting old). Anyways, after spending a lot of time and effort to determine what work would be required to migrate our software and producing some prototype code, Microsoft came out with this new fangled NT thing which just ran our existing binaries (and probably still does if I bothered to try them) and the fortunes of Microsoft and IBM have diverged ever since.

Anyways, even later on Microsoft was very keen on having third parties participate in their BETA testing ? and that hasn?t changed much based on the availability of the plethora of RC builds on MSDN and even publically. I suspense it makes sense as it is cost effective to have others test and validate your software if they are willing to volunteer and there are good reasons why we might want to have some idea of what those in Redmond are planning for us. Anyways I seem to have gotten over my nightmares of working with BETA OS versions ? although I?m sure the scars have made me a better programmer on balance.

As I said, the best thing I got out of this was a triangular prism of plastic with the OS/2 logo suspended partway through. It was a partners only thing ??

Sent from Mailhttps: for Windows 10

From: xxxxx@hotmail.commailto:xxxxx
Sent: February 9, 2017 7:45 AM
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: RE:[ntdev] Re: MmGetMdlByteCount greater than TransferBufferLength

>Many people did. The development of what was originally called NT OS/2 began in 1988 or 1989

Well, normally I would expect them to be mainly on the development team, but, OTOH, any OS
needs some hardware to run on, and,hence, some third-party drivers may be included in the very first OS release. Therefore, a bit of OS work may be,indeed, done by the third-party developers even prior to its first official release…

Anton Bassov


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></mailto:xxxxx></mailto:xxxxx></https:>

I have once read that the guys who wrote NT came from DEC. I think it was about the Windows PE binary format. This format was said to be used in VAX computers and brought to Windows NT by DEC engineers.

Mark Russinovich wrote a very interesting article about Windows NT’s history (in 1998). Every windows driver writer should read it.

http://m.windowsitpro.com/windows-client/windows-nt-and-vms-rest-story