Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Home NTDEV

Before Posting...

Please check out the Community Guidelines in the Announcements and Administration Category.

More Info on Driver Writing and Debugging


The free OSR Learning Library has more than 50 articles on a wide variety of topics about writing and debugging device drivers and Minifilters. From introductory level to advanced. All the articles have been recently reviewed and updated, and are written using the clear and definitive style you've come to expect from OSR over the years.


Check out The OSR Learning Library at: https://www.osr.com/osr-learning-library/


DeviceIoControl buffer in kernel

OSR_Community_UserOSR_Community_User Member Posts: 110,217
edited October 2023 in NTDEV
Hi,



The buffer sent from user space using DeviceIoControl is eventually
getting to kernel as a non paged pool buffer.

However, does anybody know if this buffer is also physically contiguous
? If not, is there a way to force it to be ?



Thanks.
Post edited by Peter_Viscarola_(OSR) on

Comments

  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    The received buffer may or may not be physically contiguous. There is no way to force it to be physically contiguous.

    --Mark Cariddi
    OSR, Open Systems Resources, Inc.

    From: [email protected] [mailto:[email protected]] On Behalf Of [email protected]
    Sent: Thursday, January 13, 2011 9:02 AM
    To: Windows System Software Devs Interest List
    Subject: [ntdev] DeviceIoControl buffer in kernel

    Hi,

    The buffer sent from user space using DeviceIoControl is eventually getting to kernel as a non paged pool buffer.
    However, does anybody know if this buffer is also physically contiguous ? If not, is there a way to force it to be ?

    Thanks.


    ---
    NTDEV is sponsored by OSR

    For our schedule of WDF, WDM, debugging and other seminars visit:
    http://www.osr.com/seminars

    To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
  • Don_Burn_1Don_Burn_1 Member Posts: 4,311
    No it is not physically contiguous, and assuming you are talking
    BUFFERED_IO it probably isn't too smart to make it more than 4KB since
    that is the size where DIRECT_IO starts becoming faster. AFAIK there
    is no way to force user space allocations to be physically contiguous.
    Requiring contiguous physical memory is a bad idea even in the kernel,
    since it is too easy to get fragmented not be able to get a buffer when
    needed.


    Don Burn (MVP, Windows DKD)
    Windows Filesystem and Driver Consulting
    Website: http://www.windrvr.com
    Blog: http://msmvps.com/blogs/WinDrvr



    "[email protected]" <[email protected]> wrote in message
    news:xxxxx@ntdev:

    > Hi,
    >
    >
    >
    > The buffer sent from user space using DeviceIoControl is eventually
    > getting to kernel as a non paged pool buffer.
    >
    > However, does anybody know if this buffer is also physically contiguous
    > ? If not, is there a way to force it to be ?
    >
    >
    >
    > Thanks.
  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 9,160
    <QUOTE>
    The buffer sent from user space using DeviceIoControl is eventually
    getting to kernel as a non paged pool buffer.
    </QUOTE>

    Well, that's true depending on the transfer type you specify as part of the IOCTL definition. It MIGHT be, or it might NOT be. If the IOCTL uses METHOD_BUFFERED, the buffer will be in non-paged pool. If the IOCTL uses METHOD_IN_DIRECT, then the original requestor's data buffer will be used.

    IF the IOCTL uses METHOD_BUFFERED, the buffer may or may not be physically contiguous, as Mark said.

    But the REAL question is: Why would you care? It's hard to think of a legitimate reason you'd want to know if the buffer used in a METHOD_BUFFERED IOCTL is physically contiguous. Perhaps you'll tell us what it is you're trying to do, and why you care about physical contiguity?

    <QUOTE>
    assuming you are talking BUFFERED_IO it probably isn't too smart to make it more than 4KB since
    that is the size where DIRECT_IO starts becoming faster.
    </QUOTE>

    Hmmmm... I'm afraid I have to differ with Mr. Burn here, at least slightly. The 1 page trade-off certainly WAS correct in the past. THESE days, however, with much faster CPU speeds, it's hard to really know WHAT the trade-off is between METHOD_BUFFERED and METHOD_IN_DIRECT. And the trade-off is, in fact, a rather complex one: METHOD_BUFFERED impacts CPU time in the context of the thread, whereas METHOD_xxx_DIRECT impacts overall system performance in terms of TLB invalidations. So, like so many things... when it comes to deciding what the maximum appropriate size is for an IOCTL buffer before moving from buffered to direct, the only real answer is "it depends."

    Peter
    OSR

    Peter Viscarola
    OSR
    @OSRDrivers

  • Alex_GrigAlex_Grig Member Posts: 3,238
    Why would method_direct impact system performance in terms of TLB invalidations? Only when you don't do DMA and map the buffer to the system space instead? When the buffer is being mapped, no invalidation is necessary; only when it's unmapped, at which time it's done (in the latest kernels, at least) lazily in batches.
  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 9,160
    <QUOTE>
    Only when you don't do DMA and map the buffer to the system space instead?
    </QUOTE>

    Yes... which would be the only time you'd be comparing Direct with Buffered, right?

    <QUOTE>
    When the buffer is being mapped, no invalidation is necessary; only when it's
    unmapped, at which time it's done (in the latest kernels, at least) lazily in
    batches.
    </QUOTE>

    Yup. But it's still has to be done, right?

    Peter
    OSR

    Peter Viscarola
    OSR
    @OSRDrivers

  • G.Simon_SelvarajG.Simon_Selvaraj Member Posts: 31
    The MmAllocateContiguousMemory routine allocates a range of physically contiguous, nonpaged memory and maps it to the system address space.

    PVOID
    MmAllocateContiguousMemory(
    __in SIZE_T NumberOfBytes,
    __in PHYSICAL_ADDRESS HighestAcceptableAddress
    );
  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 9,160
    OK... but sorry... why is that relevant to this thread? I must be missing your point.

    Peter
    OSR

    Peter Viscarola
    OSR
    @OSRDrivers

  • Maxim_S._ShatskihMaxim_S._Shatskih Member Posts: 10,396
    >However, does anybody know if this buffer is also physically contiguous ?

    No.

    The only physically contiguous buffers are the ones allocated by ->AllocateCommonBuffer or HalAllocateCommonBuffer. And, if there is an IOMMU, then even these buffers are not such - they are bus-side-contiguous.

    Windows is built under a strong idea that "DMA is the only need for physically contiguous buffers".

    There are MmAllocateContiguous/NonCachedMemory calls, but they are only intended for DMA adapter object writers to implement ->AllocateCommonBuffer. Using them in your own code to implement a DMA common buffer is a wrong idea - you don't know the correct requirements and thus the correct parameter values for the Mm call, only the adapter object (owned by pci.sys usually) knows. So, if you use Mm function directly, then your code will probably break dependent on IOMMU/lack of IOMMU and such.

    Hardware IOMMUs are rare (AGP GART is maybe the only wide-used implementation for usual PCs), but there were some rumours that, due to major MS's effort of "Windows must be a good guest OS", they have probably added IOMMU support to Windows, for hardware-less VM case, just to simplify creation of emulated hardware in the guest, to simplify the DMA engine, since the guest OS is assumed to support IOMMU.

    So, use the DMA APIs - IoMapTransfer, ->GetScatterGatherList etc. They will map any MDL to DMA space and give you the SGL, you will just need to convert it to the format of your hardware and submit.

    --
    Maxim S. Shatskih
    Windows DDK MVP
    [email protected]
    http://www.storagecraft.com
  • This is a very old thread, but I found an information that might be helpful. On MSFT page there is info that when the BUFFERED I/O is used the I/O manager allocates a contigous memory from non-paged pool, but using BUFFERED I/O for large transfer is not recommended since such large allocation may fail:

    https://learn.microsoft.com/en-us/windows-hardware/drivers/kernel/using-buffered-i-o :

    Drivers that transfer large amounts of data at a time, in particular, drivers that do multipage transfers, should not attempt to use buffered I/O. As the system runs, nonpaged pool can become fragmented so that the I/O manager cannot allocate large, contiguous system-space buffers to send in IRPs for such a driver.


    I have debugged existing BUFFERED I/O IOCTL on Win11 and confirmed that the memory allocated for DataBuffer is indeed physically contigous.

    --
    Wojciech Chojnowski
    System Software Architect

  • Tim_RobertsTim_Roberts Member - All Emails Posts: 14,832

    I have debugged existing BUFFERED I/O IOCTL on Win11 and confirmed that the memory allocated for DataBuffer is indeed physically contigous.

    Hold on. I don't believe that. Either you were misled, or you are unclear about the difference between physically contiguous and virtually contiguous. A single allocation is ALWAYS virtually contiguous, by definition, but physical space gets fragmented very quickly. Unless the system was within a few minutes of booting, no one even tries to make those buffers physically contiguous. As you say, the typical buffered ioctl buffer is only a page or two, so perhaps you might accidentally see a contiguous pair, but NO ONE makes any kind of promise about that.

    Also, this site strongly discourages "necroposting" -- resurrecting long dead threads.

    Tim Roberts, [email protected]
    Software Wizard Emeritus

  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 9,160

    Let me enlighten the necroposter, before I lock the thread:

    The memory that is used for METHOD_BUFFERD IOCTLs is not now, and has never been, guaranteed to be physically contiguous. It might be as a matter of coincidence. It will always be virtually contiguous.

    In addition, let me add that whether the memory is physically contiguous or not is not helpful, useful, or even relevant. You can't get its physical address and DMA to it. You'll need to build an MDL and push it through the Windows DMA API no matter what.

    So... definitively... no. Not guaranteed to be physically contiguous. Just as Mr. Roberts correctly asserts.

    Peter

    This necroposted thread is locked.

    Peter Viscarola
    OSR
    @OSRDrivers

This discussion has been closed.

Howdy, Stranger!

It looks like you're new here. Sign in or register to get started.

Upcoming OSR Seminars
OSR has suspended in-person seminars due to the Covid-19 outbreak. But, don't miss your training! Attend via the internet instead!
Kernel Debugging 13-17 May 2024 Live, Online
Developing Minifilters 1-5 Apr 2024 Live, Online
Internals & Software Drivers 11-15 Mar 2024 Live, Online
Writing WDF Drivers 20-24 May 2024 Live, Online