Re: MmGetMdlByteCount greater than TransferBufferLength

You keep confusing two things. An MDL and an I/O request.
An MDL just describes some memory.
You do I/O with whatever the I/O request says. If it has Length and Address fields, you use these fields. If it has an MDL, you pass the MDL to GetScatterGatherList, but still use URB/SRB length and address in GetScatterGatherList call.

And it keeps going…

On Wed, Feb 8, 2017 at 9:35 AM wrote:

> You keep confusing two things. An MDL and an I/O request.
> An MDL just describes some memory.
> You do I/O with whatever the I/O request says. If it has Length and
> Address fields, you use these fields. If it has an MDL, you pass the MDL to
> GetScatterGatherList, but still use URB/SRB length and address in
> GetScatterGatherList call.
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: <
> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

xxxxx@hotmail.com wrote:

@Tim The MSDN documentation on the MDL buffer length, and the URB transfer buffer length.

https://msdn.microsoft.com/en-us/library/windows/hardware/ff554530(v=vs.85).aspx "MmGetMdlByteCount returns the length, in bytes, of the buffer described by Mdl. "

That’s true.

https://msdn.microsoft.com/en-us/library/windows/hardware/ff540352(v=vs.85).aspx “TransferBufferLength
specifies the length, in bytes, of the buffer specified in TransferBuffer or described in TransferBufferMDL”

I will grant you that this could be worded more precisely. It should
say that TransferBufferLength specifies the length, in bytes, of the
buffer to be used for this transfer.

I assume you can understand the difficulty here. I can allocate a large
buffer, and then use parts of that buffer for different purposes. When
you get a TransferBuffer in an URB, the length is not carried along, so
there is no conflict. When you get a TransferBufferMDL, it just so
happens that the MDL carries knowledge of its own length, but that
length is not relevant to this transfer.

Your comments regarding my example merely compound the lack of documentation. You do resalise what you are saying here Tim, that it depends on who created the MDL-Irp-URB as to whether the two byte count values are the same or not?

Right. It absolutely does. Different driver stacks have different
contracts, largely because of history. That’s just the way it is.

Really? And the code is supposed to know this is it? How about if an upper filter is put between between your driver that trusts the IoManager, and suddenly your driver hasa byte count thats wrong?

BSOD.

Correct. That’s what happens when a driver violates the contract, which
is what such a filter driver would have done. An upper-level driver is
ALLOWED to assume conditions from I/O Manager. Any upper filter must
maintain those conditions.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

In fact they even had a partner program for application software. The highlight of which for me was getting a fancy paperweight with the OS/2 logo suspended in a triangular prism of plastic

Sent from Mailhttps: for Windows 10

From: Tim Robertsmailto:xxxxx
Sent: February 6, 2017 2:32 PM
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: Re: [ntdev] Re: MmGetMdlByteCount greater than TransferBufferLength

xxxxx@hotmail.com wrote:
> > This is just the way Windows architecture has been, since… well… at least 1992
>> when I started working with Windows NT.
> Oh dear…
>
> You seem to have had started working with NT before it even got released…

Many people did. The development of what was originally called NT OS/2
began in 1988 or 1989.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></mailto:xxxxx></mailto:xxxxx></https:>

>Many people did. The development of what was originally called NT OS/2 began in 1988 or 1989

Well, normally I would expect them to be mainly on the development team, but, OTOH, any OS
needs some hardware to run on, and,hence, some third-party drivers may be included in the very first OS release. Therefore, a bit of OS work may be,indeed, done by the third-party developers even prior to its first official release…

Anton Bassov

Anton

For OS/2, they had programs for all kinds of ISVs. They wanted to have applications that actually ran on the new OS when it launched , so they engaged a wide audience. At the time I was working on Line of Business applications for the financial industry (as I am again) rather than drivers and we were assigned a project to build an OS/2 version of our software. Our life became radially simpler once Microsoft decided to implement the Win32 subsystem (or was it called Win16 of something I forget; or I?m just getting old). Anyways, after spending a lot of time and effort to determine what work would be required to migrate our software and producing some prototype code, Microsoft came out with this new fangled NT thing which just ran our existing binaries (and probably still does if I bothered to try them) and the fortunes of Microsoft and IBM have diverged ever since.

Anyways, even later on Microsoft was very keen on having third parties participate in their BETA testing ? and that hasn?t changed much based on the availability of the plethora of RC builds on MSDN and even publically. I suspense it makes sense as it is cost effective to have others test and validate your software if they are willing to volunteer and there are good reasons why we might want to have some idea of what those in Redmond are planning for us. Anyways I seem to have gotten over my nightmares of working with BETA OS versions ? although I?m sure the scars have made me a better programmer on balance.

As I said, the best thing I got out of this was a triangular prism of plastic with the OS/2 logo suspended partway through. It was a partners only thing ??

Sent from Mailhttps: for Windows 10

From: xxxxx@hotmail.commailto:xxxxx
Sent: February 9, 2017 7:45 AM
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: RE:[ntdev] Re: MmGetMdlByteCount greater than TransferBufferLength

>Many people did. The development of what was originally called NT OS/2 began in 1988 or 1989

Well, normally I would expect them to be mainly on the development team, but, OTOH, any OS
needs some hardware to run on, and,hence, some third-party drivers may be included in the very first OS release. Therefore, a bit of OS work may be,indeed, done by the third-party developers even prior to its first official release…

Anton Bassov


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></mailto:xxxxx></mailto:xxxxx></https:>

I have once read that the guys who wrote NT came from DEC. I think it was about the Windows PE binary format. This format was said to be used in VAX computers and brought to Windows NT by DEC engineers.

Mark Russinovich wrote a very interesting article about Windows NT’s history (in 1998). Every windows driver writer should read it.

http://m.windowsitpro.com/windows-client/windows-nt-and-vms-rest-story

> I have once read that the guys who wrote NT came from DEC.I have once read that

the guys who wrote NT came from DEC.

This is true - NT chief architect came from DEC and had previously authored VMS. Therefore, I suspect quite a few engineers must have followed him. In fact, quite a few of this list" old timers"
come from DEC/VMS background. Our host is just one example. Another example is my “buddy” Dan Kyler - he even came up once with his “only proper spinlock implementation in existence”(which, in his opinion, happens to be a tight loop of interlocked operations) in VAX assembly language

I think it was about the Windows PE binary format.

Well, PE is based upon COFF file format, which had been used by quite a few systems - VMS happened (or, probably still happens) to be just one of them. However, architectural similarities
between VMS and NT go well beyond that - after all,where do you think an abomination known as pageable kernel code is rooted? As they say, increment every letter of 'VMS", and see what you will get…

Anton Bassov

>Our host is just one example.

Amazing ! In 1993, we were at the university, learning UNIX on computers running … VAX!

Read some old books on VMS and VMS file systems. This is where NT came
from. I think I still have a copy of the book VMS File System. It gives you
a good idea on where IRPs came from and how things tie together. We’ve come
a long way; sorta.

On Fri, Feb 10, 2017, 12:54 PM wrote:

> >Our host is just one example.
>
> Amazing ! In 1993, we were at the university, learning UNIX on computers
> running … VAX!
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: <
> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

>Read some old books on VMS and VMS file systems. This is where NT came
from. I think I still have a copy of the book VMS File System.

Thank you for the suggestion.

We’re not going to comprehensively review Windows NT history here.

The Windows NT project started on November 7th 1988, when a group of former DEC guys (7 of them, IIRC) started at MSFT. This was subsequent to a new OS (Mica) for entirely new silicon (PRISM) both then under development at DEC being cancelled.

Windows NT is remarkably like Mica (I know this as I was at DEC and privileged to have access to the Mica design documents at the time). This was litigated and settled amicably between DEC and Microsoft. Interestingly, the Alpha processor was also remarkably like PRISM. And part of the “amicable settlement” was MSFT supporting NT on Alpha.

The rest is history. Very interesting history, but history nonetheless. I swear someday I’m going to write a book about this.

There are significant architectural similarities between VMS and NT, and even between RSX-11M (another Culter OS) and NT. These reflect Cutler’s approach to OS design. He is an intensely practical engineer and just has always done things certain ways that he knows “work.”

Peter
OSR
@OSRDrivers

P.S. Yes, Anton. I was writing NT drivers well before NT was released. Lots and lots of people were. The beta was open and widely accessible.

On Fri, 10 Feb 2017, xxxxx@osr.com wrote:

We’re not going to comprehensively review Windows NT history here.

For they who want deeper information I can recomend the book
“Showstopper!” by G. Pascal Zachary, it is to a part a biography over
Dave Cutler and then describes the project at Microsoft untill the first
release of Windows NT.

Bo Branten

I’m not a fan. I found this book to be poorly written, by an author that seemed to have little or no gasp of OS architecture or development.

Then again, I haven’t looked at it since I read it when it was first released. I found it VERY disappointing. As I remember it, it spoke to few of the questions or issues I was interested in reading about.

Peter
OSR
@OSRDrivers

@ Tim “I will grant you that this could be worded more precisely.”

Fianlly!

Right, so how do we go about getting this changed. The MDL byte count should be be documented as the “total size of the MDL buffer which might be greater than the length of data to be transfered in the buffer”.

@ Alex Grigg & James Kirby. “You keep confusing two things. An MDL and an I/O request”

Sigh… No, one again, I have no problem with the way they are being used but i DO have a problem with the lack of dicumentation describing how they are being used.

Is it really so difficult to understand the difference?

>Right, so how do we go about getting this changed. The MDL byte count should be be documented

as the “total size of the MDL buffer which might be greater than the length of data to be transferred
in the buffer”.

Did you try clicking the link at the bottom of the MSDN API doc page that says “Send comments about this topic to Microsoft”. This is usually what I do when I find a doc error.

Jan

I have tried that in the past Jan, would you expect a response?

I’ve been working with the NT kernel since 3.51. Not once have I been
confused with respect to buffer sizes vs. request sizes; regardless of any
documentation inaccuracies. It all seems so obvious and pedestrian. Not to
mention that this thread has generated 40+ responses. Wowza!

On Tue, Feb 14, 2017 at 12:50 PM wrote:

> I have tried that in the past Jan, would you expect a response?
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: <
> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

xxxxx@hotmail.com wrote:

I have tried that in the past Jan, would you expect a response?

No. Generally, you get an automated response saying “thanks for your
comments”, but you won’t get anything further.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.