User data buffer copy in the kernel - Query

DDK has the sufficient documentation for this.

Kernel-mode driver architecture->Design Guide->Input/Output
Techniques->Methods for accessing data buffers->Using neither buffered
nor direct-io.

Harish


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Monisha Barooah
Sent: Sunday, April 02, 2006 9:18 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] User data buffer copy in the kernel - Query

Hi Everybody,

This was a query relating to the buffer copy of the user data made by
the windows kernel and a way to avoid it.

Direct I/O mechanism can be used for the same but this holds good
only for the ‘outbuffer’ corresponding to the ‘DeviceIoControl’ API. The
kernel copy of the user data buffer for the ‘inbuffer’ corresponding to
the ‘DeviceIoControl’ API cannot be avoided through the Direct I/O
method.

Is there any way by means of which, the kernel buffer copy
corresponding to the ‘inbuffer’ of the ‘DeviceIoControl’ API be avoided?

We need to initiate huge data transfers to the device from our
application via our custom windows driver and wanted to optimize on the
buffer copy overhead involved with the ‘inbuffer’ of the
‘DeviceIoControl’ API.

Please help provide the info if any.

Thanks,

Monisha.

=====
Monisha Barooah
Samsung India Software Operations Pvt. Ltd.
Bagmane Lakeview, Block ‘B’,
No. 66/1, Bagmane Tech Park,
C V Raman Nagar, Byrasandra,
Bangalore - 560093
Phone - 41819999, Extn:7155
Email: xxxxx@samsung.com


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Before you go down the method neither route, how big is your input buffer? If it is less then 1K, forget about the overhead of the input buffer copy, it is insignificant. I don’t think that this micro optimization is worth the extra work (in terms of security and complexity) that you will incur when method neither is used.

d


From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Harish Arora
Sent: Sunday, April 02, 2006 10:08 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

DDK has the sufficient documentation for this.
Kernel-mode driver architecture->Design Guide->Input/Output Techniques->Methods for accessing data buffers->Using neither buffered nor direct-io.

Harish


From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Monisha Barooah
Sent: Sunday, April 02, 2006 9:18 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] User data buffer copy in the kernel - Query

Hi Everybody,
?? This was a query relating to the buffer copy of the user data made by the windows kernel and a way to avoid it.
?
???Direct I/O mechanism can be used for the same but this holds good only for the ‘outbuffer’ corresponding to the ‘DeviceIoControl’ API. The?kernel?copy of the user data buffer?for the ‘inbuffer’ corresponding to the ‘DeviceIoControl’ API cannot be avoided through the Direct I/O method.
?
?? Is there any way by means of which, the kernel buffer copy?corresponding to the ‘inbuffer’?of the ‘DeviceIoControl’ API?be avoided?
?
? ?We need to initiate huge data transfers to the device from our application?via our custom windows driver and wanted to optimize on the buffer copy overhead involved with the ‘inbuffer’ of the ‘DeviceIoControl’ API.
?
???Please help provide the info if any.
?
Thanks,
Monisha.
???

Monisha Barooah
Samsung India Software Operations Pvt. Ltd.
Bagmane Lakeview, Block ‘B’,
No. 66/1, Bagmane Tech Park,
C V Raman Nagar, Byrasandra,
Bangalore - 560093
Phone - 41819999, Extn:7155
Email: xxxxx@samsung.com


Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Re-read the IOCTL documentation. ‘OutBuffer’ can represent direct IO
transfers in either direction. The transfer size is limited to ‘a little
less than 64MB’. If your ‘huge’ transfers are greater than ~64MB then you
need to look into the alternate methods discussed in this thread.

=====================
Mark Roddy DDK MVP
Windows 2003/XP/2000 Consulting
Hollis Technology Solutions 603-321-1032
www.hollistech.com


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Monisha Barooah
Sent: Monday, April 03, 2006 12:18 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] User data buffer copy in the kernel - Query

Hi Everybody,
This was a query relating to the buffer copy of the user data made by the
windows kernel and a way to avoid it.

Direct I/O mechanism can be used for the same but this holds good only
for the ‘outbuffer’ corresponding to the ‘DeviceIoControl’ API. The kernel
copy of the user data buffer for the ‘inbuffer’ corresponding to the
‘DeviceIoControl’ API cannot be avoided through the Direct I/O method.

Is there any way by means of which, the kernel buffer copy corresponding
to the ‘inbuffer’ of the ‘DeviceIoControl’ API be avoided?

We need to initiate huge data transfers to the device from our
application via our custom windows driver and wanted to optimize on the
buffer copy overhead involved with the ‘inbuffer’ of the ‘DeviceIoControl’
API.

Please help provide the info if any.

Thanks,
Monisha.

=====
Monisha Barooah
Samsung India Software Operations Pvt. Ltd.
Bagmane Lakeview, Block ‘B’,
No. 66/1, Bagmane Tech Park,
C V Raman Nagar, Byrasandra,
Bangalore - 560093
Phone - 41819999, Extn:7155
Email: xxxxx@samsung.com


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Hi Mark,

Perhaps you could answer something that has puzzled me. The ‘little less
than 64MB’ you mentioned I assume is because of the size limit on MDL’s. The
thing that puzzles me is on 64-bit Windows, the SIZE field in an MDL is
still 16-bits, and since each PFN value is now 64-bits wide instead of
32-bits wide, a 64-bit Windows MDL can only have half as many pages as a
Win32 MDL. On x64 Win64, pages are still 4k, so a Win64 MDL can only
describe about 32 MB. Am I just not reading the ddk headers correctly, or is
the transfer size limit on Win64 half the size as on Win32? I just can’t
believe this is so.

  • Jan

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark Roddy
Sent: Monday, April 03, 2006 4:31 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

Re-read the IOCTL documentation. ‘OutBuffer’ can represent direct IO
transfers in either direction. The transfer size is limited to ‘a little
less than 64MB’. If your ‘huge’ transfers are greater than ~64MB then you
need to look into the alternate methods discussed in this thread.

Good question. I have no idea what the answer is. Perhaps somebody else
will volunteer. The 16bit limit of the size field in the MDL has always
been a bit of an odd design decision. Enforcement a maximum MDL
descriptor policy should use a more abstract mechanism.


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jan Bottorff
Sent: Monday, April 03, 2006 9:32 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

Hi Mark,

Perhaps you could answer something that has puzzled me. The ‘little less
than 64MB’ you mentioned I assume is because of the size limit on MDL’s.
The thing that puzzles me is on 64-bit Windows, the SIZE field in an MDL
is still 16-bits, and since each PFN value is now 64-bits wide instead
of 32-bits wide, a 64-bit Windows MDL can only have half as many pages
as a Win32 MDL. On x64 Win64, pages are still 4k, so a Win64 MDL can
only describe about 32 MB. Am I just not reading the ddk headers
correctly, or is the transfer size limit on Win64 half the size as on
Win32? I just can’t believe this is so.

  • Jan

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark Roddy
Sent: Monday, April 03, 2006 4:31 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

Re-read the IOCTL documentation. ‘OutBuffer’ can represent direct IO
transfers in either direction. The transfer size is limited to ‘a little
less than 64MB’. If your ‘huge’ transfers are greater than ~64MB then
you need to look into the alternate methods discussed in this thread.


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

I don’t believe the 16-bit size field in the MDL header affects the
maximum size of the MDL anymore.

-p


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Roddy, Mark
Sent: Monday, April 03, 2006 7:55 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

Good question. I have no idea what the answer is. Perhaps somebody else
will volunteer. The 16bit limit of the size field in the MDL has always
been a bit of an odd design decision. Enforcement a maximum MDL
descriptor policy should use a more abstract mechanism.


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jan Bottorff
Sent: Monday, April 03, 2006 9:32 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

Hi Mark,

Perhaps you could answer something that has puzzled me. The ‘little less
than 64MB’ you mentioned I assume is because of the size limit on MDL’s.
The thing that puzzles me is on 64-bit Windows, the SIZE field in an MDL
is still 16-bits, and since each PFN value is now 64-bits wide instead
of 32-bits wide, a 64-bit Windows MDL can only have half as many pages
as a Win32 MDL. On x64 Win64, pages are still 4k, so a Win64 MDL can
only describe about 32 MB. Am I just not reading the ddk headers
correctly, or is the transfer size limit on Win64 half the size as on
Win32? I just can’t believe this is so.

  • Jan

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark Roddy
Sent: Monday, April 03, 2006 4:31 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

Re-read the IOCTL documentation. ‘OutBuffer’ can represent direct IO
transfers in either direction. The transfer size is limited to ‘a little
less than 64MB’. If your ‘huge’ transfers are greater than ~64MB then
you need to look into the alternate methods discussed in this thread.


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Then the published header files (ntddk.h and ntifs.h) are at best
misleading. From W2K3SP1 kit:

#define MmInitializeMdl(MemoryDescriptorList, BaseVa, Length) { \

(MemoryDescriptorList)->Next = (PMDL) NULL; \

(MemoryDescriptorList)->Size = (CSHORT)(sizeof(MDL) + \

(sizeof(PFN_NUMBER) *
ADDRESS_AND_SIZE_TO_SPAN_PAGES((BaseVa), (Length)))); \

(MemoryDescriptorList)->MdlFlags = 0; \

(MemoryDescriptorList)->StartVa = (PVOID) PAGE_ALIGN((BaseVa)); \

(MemoryDescriptorList)->ByteOffset = BYTE_OFFSET((BaseVa)); \

(MemoryDescriptorList)->ByteCount = (ULONG)(Length); \

}

This clearly does set the size field of the MDL based upon the number of
pages being described. It is certainly possible that Mm no longer cares
about this field, but in that case it might be wise to at least change
this macro so that it indicates “don’t care” in some meaningful way
(e.g., I’d suggest setting it to zero, or any value < sizeof(MDL) since
those would be below any previously legal range.) It is also possible
that this is one of the myriad of changes that are present in an as yet
unreleased version of Windows (and thus subject to change) - in which
case the OP probably still needs to worry about this for the next 5-10
years while Windows XP and W2K3 continue to be viable platforms.

Also, I’m curious how MmBuildPartialMdl works in a world where Mm has no
idea how large the MDL is - the DDK documentation says “The MDL must be
large enough to map the subrange specified by VirtualAddress and Length”
which to me means someone must know how big the MDL is. If I’m
constructing a driver that re-uses MDLs it would seem this now requires
a superstructure containing my MDL + the OS MDL so I can keep track of
the size (or I could just allocate everything to be “as big as I might
ever need” which in the world from whence I come would mean “big enough
to describe a 4GB buffer…”) Absent the ability to validate this
information, it would seem this is an invitation to buffer overrun bugs.

Regards,

Tony

Tony Mason

Consulting Partner

OSR Open Systems Resources, Inc.

http://www.osr.com


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Peter Wieland
Sent: Monday, April 03, 2006 12:14 PM
To: ntdev redirect
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

I don’t believe the 16-bit size field in the MDL header affects the
maximum size of the MDL anymore.

-p


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Roddy, Mark
Sent: Monday, April 03, 2006 7:55 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

Good question. I have no idea what the answer is. Perhaps somebody else
will volunteer. The 16bit limit of the size field in the MDL has always
been a bit of an odd design decision. Enforcement a maximum MDL
descriptor policy should use a more abstract mechanism.


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jan Bottorff
Sent: Monday, April 03, 2006 9:32 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

Hi Mark,

Perhaps you could answer something that has puzzled me. The ‘little less
than 64MB’ you mentioned I assume is because of the size limit on MDL’s.
The thing that puzzles me is on 64-bit Windows, the SIZE field in an MDL
is still 16-bits, and since each PFN value is now 64-bits wide instead
of 32-bits wide, a 64-bit Windows MDL can only have half as many pages
as a Win32 MDL. On x64 Win64, pages are still 4k, so a Win64 MDL can
only describe about 32 MB. Am I just not reading the ddk headers
correctly, or is the transfer size limit on Win64 half the size as on
Win32? I just can’t believe this is so.

  • Jan

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark Roddy
Sent: Monday, April 03, 2006 4:31 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

Re-read the IOCTL documentation. ‘OutBuffer’ can represent direct IO
transfers in either direction. The transfer size is limited to ‘a little
less than 64MB’. If your ‘huge’ transfers are greater than ~64MB then
you need to look into the alternate methods discussed in this thread.


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

I remember hearing this from Landy. I don’t remember the details
though. I’ll check.

You’ve shown how to compute the size of an MDL from its contents. And
IoBuildPartialMdl returns VOID, so it’s not going to be doing any
validity checking anyway (except, perhaps, if the verifier is enabled).

-p


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Monday, April 03, 2006 9:34 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

Then the published header files (ntddk.h and ntifs.h) are at best
misleading. From W2K3SP1 kit:

#define MmInitializeMdl(MemoryDescriptorList, BaseVa, Length) { \

(MemoryDescriptorList)->Next = (PMDL) NULL; \

(MemoryDescriptorList)->Size = (CSHORT)(sizeof(MDL) + \

(sizeof(PFN_NUMBER) *
ADDRESS_AND_SIZE_TO_SPAN_PAGES((BaseVa), (Length)))); \

(MemoryDescriptorList)->MdlFlags = 0; \

(MemoryDescriptorList)->StartVa = (PVOID) PAGE_ALIGN((BaseVa)); \

(MemoryDescriptorList)->ByteOffset = BYTE_OFFSET((BaseVa)); \

(MemoryDescriptorList)->ByteCount = (ULONG)(Length); \

}

This clearly does set the size field of the MDL based upon the number of
pages being described. It is certainly possible that Mm no longer cares
about this field, but in that case it might be wise to at least change
this macro so that it indicates “don’t care” in some meaningful way
(e.g., I’d suggest setting it to zero, or any value < sizeof(MDL) since
those would be below any previously legal range.) It is also possible
that this is one of the myriad of changes that are present in an as yet
unreleased version of Windows (and thus subject to change) - in which
case the OP probably still needs to worry about this for the next 5-10
years while Windows XP and W2K3 continue to be viable platforms.

Also, I’m curious how MmBuildPartialMdl works in a world where Mm has no
idea how large the MDL is - the DDK documentation says “The MDL must be
large enough to map the subrange specified by VirtualAddress and Length”
which to me means someone must know how big the MDL is. If I’m
constructing a driver that re-uses MDLs it would seem this now requires
a superstructure containing my MDL + the OS MDL so I can keep track of
the size (or I could just allocate everything to be “as big as I might
ever need” which in the world from whence I come would mean “big enough
to describe a 4GB buffer…”) Absent the ability to validate this
information, it would seem this is an invitation to buffer overrun bugs.

Regards,

Tony

Tony Mason

Consulting Partner

OSR Open Systems Resources, Inc.

http://www.osr.com


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Peter Wieland
Sent: Monday, April 03, 2006 12:14 PM
To: ntdev redirect
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

I don’t believe the 16-bit size field in the MDL header affects the
maximum size of the MDL anymore.

-p


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Roddy, Mark
Sent: Monday, April 03, 2006 7:55 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

Good question. I have no idea what the answer is. Perhaps somebody else
will volunteer. The 16bit limit of the size field in the MDL has always
been a bit of an odd design decision. Enforcement a maximum MDL
descriptor policy should use a more abstract mechanism.


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jan Bottorff
Sent: Monday, April 03, 2006 9:32 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

Hi Mark,

Perhaps you could answer something that has puzzled me. The ‘little less
than 64MB’ you mentioned I assume is because of the size limit on MDL’s.
The thing that puzzles me is on 64-bit Windows, the SIZE field in an MDL
is still 16-bits, and since each PFN value is now 64-bits wide instead
of 32-bits wide, a 64-bit Windows MDL can only have half as many pages
as a Win32 MDL. On x64 Win64, pages are still 4k, so a Win64 MDL can
only describe about 32 MB. Am I just not reading the ddk headers
correctly, or is the transfer size limit on Win64 half the size as on
Win32? I just can’t believe this is so.

  • Jan

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark Roddy
Sent: Monday, April 03, 2006 4:31 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

Re-read the IOCTL documentation. ‘OutBuffer’ can represent direct IO
transfers in either direction. The transfer size is limited to ‘a little
less than 64MB’. If your ‘huge’ transfers are greater than ~64MB then
you need to look into the alternate methods discussed in this thread.


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

My point isn’t that Mm can’t be changed to work properly ignoring this
field (how else could Landy support 4GB paging I/Os,) but that it is
rather difficult for anyone ELSE to know this fact. I agree that
there’s nothing here that prevents one from computing this, but absent a
discussion in the header file and/or documentation, it’s not clear that
this is necessary. It seems like one of those subtle changes that will
break compatibility for a driver that did MDL recycling and used the
length field.

And let’s not forget the checked build - validating the size of the MDL
certainly seems like something I’d expect (and have seen in the past) in
the checked build. Perhaps other’s have observed this in the past as
well.

Finally, the only other issue would be “to what versions does this
apply”. If this is something that is only true for Vista, prudent
coding advice says “don’t use MDLs larger than 64k until my driver must
only work/run on Vista or later releases.” For many of us, that means
this remains an issue until at least 2010.

Regards,

Tony

Tony Mason

Consulting Partner

OSR Open Systems Resources, Inc.

http://www.osr.com


I remember hearing this from Landy. I don’t remember the details
though. I’ll check.

You’ve shown how to compute the size of an MDL from its contents. And
IoBuildPartialMdl returns VOID, so it’s not going to be doing any
validity checking anyway (except, perhaps, if the verifier is enabled).

While it’s true that method direct only uses direct I/O for the
outbuffer, it can be used for either data transfer into or out of the
driver (METHOD_OUT_DIRECT/METHOD_IN_DIRECT). The input buffer is then
available for whatever useful parameters you want to pass in with the
data buffer.

Robert Newton

This came up on my blog in Feb. http://blogs.msdn.com/doronh/archive/2006/02/14/532149.aspx

re: \Device\Suggestions

Friday, February 17, 2006 2:32 PM by poltrone
The size of a memory block that can be descibed by an MDL is limited by the WORD-sized field describing the size of the MDL itself. I wonder why the size field was not expanded (say to a DWORD) when the step was made to 64 bit environments.

re: \Device\Suggestions

Sunday, February 19, 2006 10:58 PM by paulsli
Poltrone, we’ve addressed this in Vista; the size field can wrap so there is no longer that limitation when calling IoAllocateMdl.

Also, hopefully you aren’t relying on the ‘internal’ size field of MDLs since it is documented as opaque :slight_smile:

re: \Device\Suggestions

Sunday, February 19, 2006 11:21 PM by doronh
More info on the Size field. I asked our memory manager dev and said:

“The Size field is ignored (was never needed from the beginning).
The real governing field is the ULONG ByteCount (hence MDLs can only describe a range < 4gb in length).”

d

– I can spell, I just can’t type.


From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Monday, April 03, 2006 10:26 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

My point isn’t that Mm can’t be changed to work properly ignoring this field (how else could Landy support 4GB paging I/Os,) but that it is rather difficult for anyone ELSE to know this fact.? I agree that there’s nothing here that prevents one from computing this, but absent a discussion in the header file and/or documentation, it’s not clear that this is necessary. It seems like one of those subtle changes that will break compatibility for a driver that did MDL recycling and used the length field.

And let’s not forget the checked build - validating the size of the MDL certainly seems like something I’d expect (and have seen in the past) in the checked build. ?Perhaps other’s have observed this in the past as well.

Finally, the only other issue would be “to what versions does this apply”.? If this is something that is only true for Vista, prudent coding advice says “don’t use MDLs larger than 64k until my driver must only work/run on Vista or later releases.”? For many of us, that means this remains an issue until at least 2010.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com


I remember hearing this from Landy.? I don’t remember the details though.? I’ll check.
?
You’ve shown how to compute the size of an MDL from its contents.? And IoBuildPartialMdl returns VOID, so it’s not going to be doing any validity checking anyway (except, perhaps, if the verifier is enabled).
?


Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Doron,

While it’s all fine and cool for you MS devs for whom the currently
shipping OS is “Legacy” to not care about limitations present in existing
OS versions, those of us who don’t live in your rarefied environment don’t
have that luxury. I’m not trying to be nasty, but I can’t rely on future
behavior in today’s OS versions. I’m sure I’m not the only one on this
list to have scars from doing just that.

The Windows Server 2003 SP1 DDK docs for IoAllocateMdl say this about the
limitation on MDL length:

ULONG Length Specifies the length in bytes of the buffer the MDL is to
describe. This value must be less than (PAGE_SIZE * (65535-sizeof(MDL)) /
sizeof(ULONG_PTR)).

So while we shouldn’t be poking into the MDL fields, we at least need to
adhere to the documented limitations. Unless the doc is wrong? Sure
would be nice if some of your SE devs would hang out here.

Assuming the doc is right, should we just be passing the result of
MmSizeOfMdl() into that parameter? Is MmSizeOfMdl() guaranteed to “do the
right thing” on every platform? Will it barf if the buffer to be
described is too big? The doc for that particular DDI is pretty sparse.

Phil

Philip D. Barila
Seagate Technology LLC
(720) 684-1842

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Doron Holan
Sent: Monday, April 03, 2006 12:15 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

This came up on my blog in Feb.
http://blogs.msdn.com/doronh/archive/2006/02/14/532149.aspx

re: \Device\Suggestions

Friday, February 17, 2006 2:32 PM by poltrone
The size of a memory block that can be descibed by an MDL is limited by
the WORD-sized field describing the size of the MDL itself. I wonder why
the size field was not expanded (say to a DWORD) when the step was made to
64 bit environments.

re: \Device\Suggestions

Sunday, February 19, 2006 10:58 PM by paulsli
Poltrone, we’ve addressed this in Vista; the size field can wrap so there
is no longer that limitation when calling IoAllocateMdl.

Also, hopefully you aren’t relying on the ‘internal’ size field of MDLs
since it is documented as opaque :slight_smile:

re: \Device\Suggestions

Sunday, February 19, 2006 11:21 PM by doronh
More info on the Size field. I asked our memory manager dev and said:

“The Size field is ignored (was never needed from the beginning).
The real governing field is the ULONG ByteCount (hence MDLs can only
describe a range < 4gb in length).”

d

– I can spell, I just can’t type.

If I remember correctly, while indeed the Size field is ignored and has been for almost ever, the IoManager enforces a restriction on MDL size at creation time (IoAllocateMdl) that corresponds to the bit width of the Size field. I suspect that this limit remains in place. It has long been possible (at least since NT4) to build one’s own MDL by hand that was way bigger than 64M, the difficult part was locking all the corresponding pages.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Doron Holan
Sent: Monday, April 03, 2006 2:15 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

This came up on my blog in Feb. http://blogs.msdn.com/doronh/archive/2006/02/14/532149.aspx

re: \Device\Suggestions

Friday, February 17, 2006 2:32 PM by poltrone
The size of a memory block that can be descibed by an MDL is limited by the WORD-sized field describing the size of the MDL itself. I wonder why the size field was not expanded (say to a DWORD) when the step was made to 64 bit environments.

re: \Device\Suggestions

Sunday, February 19, 2006 10:58 PM by paulsli
Poltrone, we’ve addressed this in Vista; the size field can wrap so there is no longer that limitation when calling IoAllocateMdl.

Also, hopefully you aren’t relying on the ‘internal’ size field of MDLs since it is documented as opaque :slight_smile:

re: \Device\Suggestions

Sunday, February 19, 2006 11:21 PM by doronh
More info on the Size field. I asked our memory manager dev and said:

“The Size field is ignored (was never needed from the beginning).
The real governing field is the ULONG ByteCount (hence MDLs can only describe a range < 4gb in length).”

d

– I can spell, I just can’t type.


From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Monday, April 03, 2006 10:26 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] User data buffer copy in the kernel - Query

My point isn’t that Mm can’t be changed to work properly ignoring this field (how else could Landy support 4GB paging I/Os,) but that it is rather difficult for anyone ELSE to know this fact.? I agree that there’s nothing here that prevents one from computing this, but absent a discussion in the header file and/or documentation, it’s not clear that this is necessary. It seems like one of those subtle changes that will break compatibility for a driver that did MDL recycling and used the length field.

And let’s not forget the checked build - validating the size of the MDL certainly seems like something I’d expect (and have seen in the past) in the checked build. ?Perhaps other’s have observed this in the past as well.

Finally, the only other issue would be “to what versions does this apply”.? If this is something that is only true for Vista, prudent coding advice says “don’t use MDLs larger than 64k until my driver must only work/run on Vista or later releases.”? For many of us, that means this remains an issue until at least 2010.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com


I remember hearing this from Landy.? I don’t remember the details though.? I’ll check.
?
You’ve shown how to compute the size of an MDL from its contents.? And IoBuildPartialMdl returns VOID, so it’s not going to be doing any validity checking anyway (except, perhaps, if the verifier is enabled).
?


Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

> We need to initiate huge data transfers to the device from our application
via

our custom windows driver and wanted to optimize on the buffer copy overhead
involved with the ‘inbuffer’ of the ‘DeviceIoControl’ API.

Use METHOD_NEITHER, and do MmProbeAndLockPages of the IN buffer in your driver
manually (under a __try/__except block).

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com