I have a PCI bus master device which supports packet based DMA up to a
packet length of 64KB. I call GetScatterGatherList to create a
SCATTER_GATHER_LIST and then create one packet transfer instruction for
each SCATTER_GATHER_ELEMENT. Can I assume that each
SCATTER_GATHER_ELEMENT describes one physical page only? Or does
GetScatterGatherList combine adjacent (and contiguous) pages together in
one SCATTER_GATHER_ELEMENT so that such an element could exceed 64K?
The DDK doc is not clear on this. Is this documented elsewhere?
Thanks.
Udo
Adjacent pages will be combined, and the length can exceed 64K if you said you could transfer more than that when you defined the adapter. At least, that’s how I remember it. Don’t know about documentation.
It’s just a bit unusual to actually see such large chunks in practice [maybe I just work too much on low-memory machines].
Udo Eberhardt wrote:
I have a PCI bus master device which supports packet based DMA up to a
packet length of 64KB. I call GetScatterGatherList to create a
SCATTER_GATHER_LIST and then create one packet transfer instruction
for each SCATTER_GATHER_ELEMENT. Can I assume that each
SCATTER_GATHER_ELEMENT describes one physical page only? Or does
GetScatterGatherList combine adjacent (and contiguous) pages together
in one SCATTER_GATHER_ELEMENT so that such an element could exceed 64K?
You can’t assume anything. In my experience, GSGL does not coalesce
adjacent pages, but I don’t believe it’s safe to assume that. And,
honestly, it’s not that hard to handle.
On the other hand, within a few minutes of boot, physical memory becomes
so fragmented that the odds against getting two consecutive physical
pages in an arbitrary user-mode buffer are astronomical.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
Thanks for clarification. I would expect that large contiguous junks
still exist immediately after boot.
Udo
Bob Kjelgaard wrote:
Adjacent pages will be combined, and the length can exceed 64K if you
said you could transfer more than that when you defined the adapter.
At least, that’s how I remember it. Don’t know about documentation.
It’s just a bit unusual to actually see such large chunks in practice
[maybe I just work too much on low-memory machines].
It *can* describe up to 4GB in one element, since the length member is a
ULONG. That would be 2^20 contiguous 4K pages. It’s not very likely that
you will have an element that describes a 64K (16 contiguous 4K pages)
buffer segment, much less 4GB, however, you should check each element for
any boundary constraints you have and break up the transfer as necessary.
Phil
Philip D. Barila
Seagate Technology LLC
(720) 684-1842
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Udo Eberhardt
Sent: Thursday, November 09, 2006 2:38 PM
To: “Windows System Software Devs Interest List”
Subject: [ntdev] MDL to SCATTER_GATHER_LIST mapping
I have a PCI bus master device which supports packet based DMA up to a
packet length of 64KB. I call GetScatterGatherList to create a
SCATTER_GATHER_LIST and then create one packet transfer instruction for
each SCATTER_GATHER_ELEMENT. Can I assume that each
SCATTER_GATHER_ELEMENT describes one physical page only? Or does
GetScatterGatherList combine adjacent (and contiguous) pages together in
one SCATTER_GATHER_ELEMENT so that such an element could exceed 64K?
The DDK doc is not clear on this. Is this documented elsewhere?
Thanks.
Udo
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tim Roberts
Sent: Thursday, November 09, 2006 2:59 PM
To: “Windows System Software Devs Interest List”
Subject: Re: [ntdev] MDL to SCATTER_GATHER_LIST mapping
> On the other hand, within a few minutes of boot, physical memory
> becomes so fragmented that the odds against getting two
> consecutive physical pages in an arbitrary user-mode buffer
> are astronomical.
That’s a bit hyperbolic, don’t you think?
My experience says that 8K and 12K segments are quite common. In fact,
when I was attempting to test the limits of my SGL (describing a 32 MB UM
buffer, so 8193 segments, plus some extra spice unique to the hardware), I
had absolute misery getting a buffer that consisted of nothing but 4K pages
plus the two partials at either end. Finally got it after enough
allocations. Took more than 40 “alloc buffer/pass buffer to IO” iterations
to do it.
Phil
Philip D. Barila
Seagate Technology LLC
(720) 684-1842
Tim Roberts wrote:
Udo Eberhardt wrote:
>for each SCATTER_GATHER_ELEMENT. Can I assume that each
>SCATTER_GATHER_ELEMENT describes one physical page only? Or does
>GetScatterGatherList combine adjacent (and contiguous) pages together
>in one SCATTER_GATHER_ELEMENT so that such an element could exceed 64K?
You can’t assume anything. In my experience, GSGL does not coalesce
adjacent pages, but I don’t believe it’s safe to assume that. And,
honestly, it’s not that hard to handle.
On the other hand, within a few minutes of boot, physical memory becomes
so fragmented that the odds against getting two consecutive physical
pages in an arbitrary user-mode buffer are astronomical.
GSGL will, at least sometimes, coaless adjacent pages. I’ve spent some
time looking at S/G lists it has generated and it does sometimes give me
multipage elements. I have not, in fact, noticed it fail to coaless
adjacent pages into one S/g entry.
And actually, I find that we quite often get two adjacent pages in our
buffers (I’ve never seen more than that, but 2 I see quite frequently).
Michael Kohne
xxxxx@kohne.org
If I can be rude - you will never find out the semantics.
Even very simple tasks have speed bumps.
And the documentation won’t help.
Sad when you paid a lot of money for the IFS kit.
Cheaper now, but not better.
The DDK is poor, at least the one they release.
The samples wouldn’t get them hired by me.
MS in house could not have written drivers with
the information they give out.
Documentation should actually reveal semantics and
interactions. But that’s not the way it is.
brucee
On 11/10/06, Michael Kohne wrote:
> Tim Roberts wrote:
> > Udo Eberhardt wrote:
> >
> >>for each SCATTER_GATHER_ELEMENT. Can I assume that each
> >>SCATTER_GATHER_ELEMENT describes one physical page only? Or does
> >>GetScatterGatherList combine adjacent (and contiguous) pages together
> >>in one SCATTER_GATHER_ELEMENT so that such an element could exceed 64K?
> >
> >
> > You can’t assume anything. In my experience, GSGL does not coalesce
> > adjacent pages, but I don’t believe it’s safe to assume that. And,
> > honestly, it’s not that hard to handle.
> >
> > On the other hand, within a few minutes of boot, physical memory becomes
> > so fragmented that the odds against getting two consecutive physical
> > pages in an arbitrary user-mode buffer are astronomical.
> >
> GSGL will, at least sometimes, coaless adjacent pages. I’ve spent some
> time looking at S/G lists it has generated and it does sometimes give me
> multipage elements. I have not, in fact, noticed it fail to coaless
> adjacent pages into one S/g entry.
>
> And actually, I find that we quite often get two adjacent pages in our
> buffers (I’ve never seen more than that, but 2 I see quite frequently).
>
> Michael Kohne
> xxxxx@kohne.org
>
>
> —
> Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>
>>And actually, I find that we quite often get two adjacent pages in our
buffers (I’ve never seen more than that, but 2 I see quite frequently).<<
Sounds like adjacent pages is more common than I remember it being. But that’s a good thing- better throughput with fewer Isrs and Dpcs swallowing up processor bandwidth. No complaint here.
>> I would expect that large contiguous junks still exist immediately after boot.
Udo<<
Good point- happens I’ve not needed to think about this in that context. Also, I was absurdly wrong in my thinking here- they should even be there later- you can certainly put a lot of memory in a machine at reasonable prices anymore. I tend to work on hand-me-downs with a lot of startup tasks and services, and test things from command prompts, so things are usually thrashed by the time I start looking.
I’m definitely guilty of incorrectly scaling a limited set of observations to a larger world view here.
Even so, there will also be a few “holes” after boot, I believe [speculatively]. If you’ve got something like a storage device loading driver images post-boot for instance, I’d expect potentially to see holes for the memory used by the initial boot loader and the discardable code and data (aka “INIT” segments) of the boot drivers [perhaps of the kernel and HAL also- I don’t recall when those segments go away]. The boot loader loads all of them into memory, then calls the kernel’s entry point, which eventually calls the entry code for all the drivers. It would be silly to not discard the init code at that time, so- offhand, anyway- I don’t see how those holes couldn’t be there. All of that has to happen before a driver using the OS support is going to begin getting asked to finish the process.
Some of the other posters probably have better insight and experience in that environment. I’ve always been interested in those early boot stages- even written toy boot loaders, tweaked partition tables, etc. But it’s a hobby, not a job. Hope to play with that sort of thing a lot more after I retire (assuming it’s still possible by then, and my eyesight holds up)- unless Halo 7 is so good I can’t put it down, anyway:).
Thanks all for your comments. I think it would be wise to create an
implementation that handles every possible case up to
SCATTER_GATHER_ELEMENTs of 4G in size. So there will be no assumption on
a particular behavior which can be implementation-specific and
change in future OS versions.
Anyway, I totally agree on the comments given by Bruce. Documentation
should be clear on such things. At least it should state that a driver
should not make assumptions on SGE sizes.
Udo