Scatter-Gather list across PCI bridge

Hello, brains. :-}
I’m trying to implement Scatter/Gather DMA on a PCI device, where the
chaining list resides in Host memory on the PC. I’m using ExAllocatePool
to allocate a chunk of memory where I will build the list. I write the
Host Address, Device Address, Length, and Next Descriptor Link (always this
one +1) for each descriptor. My hardware guy tells me that the address of
the list (and the NextDescriptor fields) is not a valid PCI address, as
seen from the device. (The Host Address does seem to be valid.)
Perhaps I should be using a different method to allocate the memory for
the list (I’m not really familiar with HalTranslateBusAddress and such, so
don’t know how that fits in…). Or maybe I need to do something extra to
the address I get from ExAllocatePool?
I’m confused. Can anyone spin me around and head me in the right
direction?
Dave Taylor
AJA Video Systems
xxxxx@aja.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You don’t mention it, but is this for NT or 2000/XP? In either case you have
to call a function that maps the transfer and converts each element of the
S/G list to a physical address and length. For 2000 this is
DMAOperations->MapTransfer and for NT 4 it is IoMapTransfer.

Gary G. Little
Staff Engineer
Broadband Storage, Inc.
xxxxx@broadstor.com

-----Original Message-----
From: xxxxx@aja.com [mailto:xxxxx@aja.com]
Sent: Tuesday, September 11, 2001 1:23 PM
To: NT Developers Interest List
Subject: [ntdev] Scatter-Gather list across PCI bridge

Hello, brains. :-}
I’m trying to implement Scatter/Gather DMA on a PCI device, where the
chaining list resides in Host memory on the PC. I’m using ExAllocatePool
to allocate a chunk of memory where I will build the list. I write the
Host Address, Device Address, Length, and Next Descriptor Link (always this
one +1) for each descriptor. My hardware guy tells me that the address of
the list (and the NextDescriptor fields) is not a valid PCI address, as
seen from the device. (The Host Address does seem to be valid.)
Perhaps I should be using a different method to allocate the memory for
the list (I’m not really familiar with HalTranslateBusAddress and such, so
don’t know how that fits in…). Or maybe I need to do something extra to
the address I get from ExAllocatePool?
I’m confused. Can anyone spin me around and head me in the right
direction?
Dave Taylor
AJA Video Systems
xxxxx@aja.com


You are currently subscribed to ntdev as: xxxxx@broadstor.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Hmmm…

Hard to know where to start with this other than to agree that you are
confused :slight_smile:

First, lets review the difference between Physical Address and Virtual
Address. Software running on the NT platform only uses virtual addresses
for memory. Memory management hardware and software translate virtual
addresses to physical addresses in order to locate the physical memory
used for any particular virtual address. The mapping from virtual to
physical can change over time.

PCI devices accessing host memory only use Physical Addresses. If you
hand a PCI device a virtual address and instruct it to perfrom some
transfer operation in or out of that address, it is very unlikely that
your request will be performed as you intended. This is why your
‘hardware guy’ was complaining about invalid PCI addresses.

When you call ExAllocatePool you obtain a virtual address of a buffer.
That virtual address maps to some number of physical page addresses.
There is no guarantee that the physical pages are contiguous (although
if you specified nonpagedpool in general they will be contiguous.) But,
nevermind that, you need a physically contiguous buffer, and you need to
use physical, not virtual, addresses for everything that your PCI device
uses.

The correct API is AllocateCommonBuffer (or if we are talking about NT4
rather than W2K HalAllocateCommonBuffer.) This API returns to you BOTH a
virtual address and a physical address (actually the documentation
refers to it as a logical address, but lets not get into that right
now.) Better yet, the api provides a guarantee that the buffer returned
is physically contiguous.

Now you can use this buffer to construct your SGL, typically as a count
plus a variable length array of {physical address, length} elements.
Inform your PCI device of the physical address of your SGL, let it do
the indexing on its own (no need for a next pointer,) and off you go.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@aja.com
Sent: Tuesday, September 11, 2001 8:23 PM
To: NT Developers Interest List
Subject: [ntdev] Scatter-Gather list across PCI bridge

Hello, brains. :-}
I’m trying to implement Scatter/Gather DMA on a PCI device,
where the
chaining list resides in Host memory on the PC. I’m using
ExAllocatePool
to allocate a chunk of memory where I will build the list. I
write the
Host Address, Device Address, Length, and Next Descriptor
Link (always this
one +1) for each descriptor. My hardware guy tells me that
the address of
the list (and the NextDescriptor fields) is not a valid PCI
address, as
seen from the device. (The Host Address does seem to be valid.)
Perhaps I should be using a different method to allocate
the memory for
the list (I’m not really familiar with HalTranslateBusAddress
and such, so
don’t know how that fits in…). Or maybe I need to do
something extra to
the address I get from ExAllocatePool?
I’m confused. Can anyone spin me around and head me in the right
direction?
Dave Taylor
AJA Video Systems
xxxxx@aja.com


You are currently subscribed to ntdev as:
xxxxx@hollistech.com To unsubscribe send a blank email to
leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Hi Mark,
I got a small about HalallocateCommonBuffer.
Will the physical address is contigouus…

If I have got 64K memory allocated by HalallocateCommonBuffer , is it
possible for me to perform 2 DMAs of 32 K each??. I mean if physical base
address is 0x10000000, for the second DMA, will the physical base will be
0x10000000 32K or will it be something different??

TIA
atma

----- Original Message -----
From: Mark Roddy
To: NT Developers Interest List
Sent: Wednesday, September 12, 2001 6:22 AM
Subject: [ntdev] RE: Scatter-Gather list across PCI bridge

> Hmmm…
>
> Hard to know where to start with this other than to agree that you are
> confused :slight_smile:
>
> First, lets review the difference between Physical Address and Virtual
> Address. Software running on the NT platform only uses virtual addresses
> for memory. Memory management hardware and software translate virtual
> addresses to physical addresses in order to locate the physical memory
> used for any particular virtual address. The mapping from virtual to
> physical can change over time.
>
> PCI devices accessing host memory only use Physical Addresses. If you
> hand a PCI device a virtual address and instruct it to perfrom some
> transfer operation in or out of that address, it is very unlikely that
> your request will be performed as you intended. This is why your
> ‘hardware guy’ was complaining about invalid PCI addresses.
>
> When you call ExAllocatePool you obtain a virtual address of a buffer.
> That virtual address maps to some number of physical page addresses.
> There is no guarantee that the physical pages are contiguous (although
> if you specified nonpagedpool in general they will be contiguous.) But,
> nevermind that, you need a physically contiguous buffer, and you need to
> use physical, not virtual, addresses for everything that your PCI device
> uses.
>
> The correct API is AllocateCommonBuffer (or if we are talking about NT4
> rather than W2K HalAllocateCommonBuffer.) This API returns to you BOTH a
> virtual address and a physical address (actually the documentation
> refers to it as a logical address, but lets not get into that right
> now.) Better yet, the api provides a guarantee that the buffer returned
> is physically contiguous.
>
> Now you can use this buffer to construct your SGL, typically as a count
> plus a variable length array of {physical address, length} elements.
> Inform your PCI device of the physical address of your SGL, let it do
> the indexing on its own (no need for a next pointer,) and off you go.
>
>
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@aja.com
> > Sent: Tuesday, September 11, 2001 8:23 PM
> > To: NT Developers Interest List
> > Subject: [ntdev] Scatter-Gather list across PCI bridge
> >
> >
> > Hello, brains. :-}
> > I’m trying to implement Scatter/Gather DMA on a PCI device,
> > where the
> > chaining list resides in Host memory on the PC. I’m using
> > ExAllocatePool
> > to allocate a chunk of memory where I will build the list. I
> > write the
> > Host Address, Device Address, Length, and Next Descriptor
> > Link (always this
> > one +1) for each descriptor. My hardware guy tells me that
> > the address of
> > the list (and the NextDescriptor fields) is not a valid PCI
> > address, as
> > seen from the device. (The Host Address does seem to be valid.)
> > Perhaps I should be using a different method to allocate
> > the memory for
> > the list (I’m not really familiar with HalTranslateBusAddress
> > and such, so
> > don’t know how that fits in…). Or maybe I need to do
> > something extra to
> > the address I get from ExAllocatePool?
> > I’m confused. Can anyone spin me around and head me in the right
> > direction?
> > Dave Taylor
> > AJA Video Systems
> > xxxxx@aja.com
> >
> > —
> > You are currently subscribed to ntdev as:
> > xxxxx@hollistech.com To unsubscribe send a blank email to
> > leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com
> >
> >
>
>
>
> —
> You are currently subscribed to ntdev as: xxxxx@tataelxsi.co.in
> To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Yes the system guarantees that both the virtual address and the
PhysicalAddress (referred to as a logical address) refer to contiguous pages
of memory.

-----Original Message-----
From: Atmacharan [mailto:xxxxx@tataelxsi.co.in]
Sent: Wednesday, September 12, 2001 8:10 AM
To: NT Developers Interest List
Subject: [ntdev] RE: Scatter-Gather list across PCI bridge

Hi Mark,
I got a small about HalallocateCommonBuffer.
Will the physical address is contigouus…

If I have got 64K memory allocated by HalallocateCommonBuffer , is it
possible for me to perform 2 DMAs of 32 K each??. I mean if physical base
address is 0x10000000, for the second DMA, will the physical base will be
0x10000000 32K or will it be something different??

TIA
atma

----- Original Message -----
From: Mark Roddy
To: NT Developers Interest List
Sent: Wednesday, September 12, 2001 6:22 AM
Subject: [ntdev] RE: Scatter-Gather list across PCI bridge

> Hmmm…
>
> Hard to know where to start with this other than to agree that you are
> confused :slight_smile:
>
> First, lets review the difference between Physical Address and Virtual
> Address. Software running on the NT platform only uses virtual
> addresses for memory. Memory management hardware and software
> translate virtual addresses to physical addresses in order to locate
> the physical memory used for any particular virtual address. The
> mapping from virtual to physical can change over time.
>
> PCI devices accessing host memory only use Physical Addresses. If you
> hand a PCI device a virtual address and instruct it to perfrom some
> transfer operation in or out of that address, it is very unlikely that
> your request will be performed as you intended. This is why your
> ‘hardware guy’ was complaining about invalid PCI addresses.
>
> When you call ExAllocatePool you obtain a virtual address of a buffer.
> That virtual address maps to some number of physical page addresses.
> There is no guarantee that the physical pages are contiguous (although
> if you specified nonpagedpool in general they will be contiguous.)
> But, nevermind that, you need a physically contiguous buffer, and you
> need to use physical, not virtual, addresses for everything that your
> PCI device uses.
>
> The correct API is AllocateCommonBuffer (or if we are talking about
> NT4 rather than W2K HalAllocateCommonBuffer.) This API returns to you
> BOTH a virtual address and a physical address (actually the
> documentation refers to it as a logical address, but lets not get into
> that right
> now.) Better yet, the api provides a guarantee that the buffer returned
> is physically contiguous.
>
> Now you can use this buffer to construct your SGL, typically as a
> count plus a variable length array of {physical address, length}
> elements. Inform your PCI device of the physical address of your SGL,
> let it do the indexing on its own (no need for a next pointer,) and
> off you go.
>
>
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@aja.com
> > Sent: Tuesday, September 11, 2001 8:23 PM
> > To: NT Developers Interest List
> > Subject: [ntdev] Scatter-Gather list across PCI bridge
> >
> >
> > Hello, brains. :-}
> > I’m trying to implement Scatter/Gather DMA on a PCI device, where
> > the chaining list resides in Host memory on the PC. I’m using
> > ExAllocatePool
> > to allocate a chunk of memory where I will build the list. I
> > write the
> > Host Address, Device Address, Length, and Next Descriptor
> > Link (always this
> > one +1) for each descriptor. My hardware guy tells me that
> > the address of
> > the list (and the NextDescriptor fields) is not a valid PCI
> > address, as
> > seen from the device. (The Host Address does seem to be valid.)
> > Perhaps I should be using a different method to allocate
> > the memory for
> > the list (I’m not really familiar with HalTranslateBusAddress
> > and such, so
> > don’t know how that fits in…). Or maybe I need to do
> > something extra to
> > the address I get from ExAllocatePool?
> > I’m confused. Can anyone spin me around and head me in the right
> > direction?
> > Dave Taylor
> > AJA Video Systems
> > xxxxx@aja.com
> >
> > —
> > You are currently subscribed to ntdev as: xxxxx@hollistech.com To
> > unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com
> >
> >
>
>
>
> —
> You are currently subscribed to ntdev as: xxxxx@tataelxsi.co.in
> To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: xxxxx@stratus.com To
unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

In our ICD, we used mmAllocateContiguousMemory to get this kind of memory,
and mmGetPhysicalAddress to convert virtual to physical. Sometimes we went
directly to the page tables, because we found it a good idea to keep API
gobbledigook within the bounds of initialization code. This was done in the
display miniport, once.

Alberto.

-----Original Message-----
From: Mark Roddy [mailto:xxxxx@hollistech.com]
Sent: Tuesday, September 11, 2001 8:53 PM
To: NT Developers Interest List
Subject: [ntdev] RE: Scatter-Gather list across PCI bridge

Hmmm…

Hard to know where to start with this other than to agree that you are
confused :slight_smile:

First, lets review the difference between Physical Address and Virtual
Address. Software running on the NT platform only uses virtual addresses
for memory. Memory management hardware and software translate virtual
addresses to physical addresses in order to locate the physical memory
used for any particular virtual address. The mapping from virtual to
physical can change over time.

PCI devices accessing host memory only use Physical Addresses. If you
hand a PCI device a virtual address and instruct it to perfrom some
transfer operation in or out of that address, it is very unlikely that
your request will be performed as you intended. This is why your
‘hardware guy’ was complaining about invalid PCI addresses.

When you call ExAllocatePool you obtain a virtual address of a buffer.
That virtual address maps to some number of physical page addresses.
There is no guarantee that the physical pages are contiguous (although
if you specified nonpagedpool in general they will be contiguous.) But,
nevermind that, you need a physically contiguous buffer, and you need to
use physical, not virtual, addresses for everything that your PCI device
uses.

The correct API is AllocateCommonBuffer (or if we are talking about NT4
rather than W2K HalAllocateCommonBuffer.) This API returns to you BOTH a
virtual address and a physical address (actually the documentation
refers to it as a logical address, but lets not get into that right
now.) Better yet, the api provides a guarantee that the buffer returned
is physically contiguous.

Now you can use this buffer to construct your SGL, typically as a count
plus a variable length array of {physical address, length} elements.
Inform your PCI device of the physical address of your SGL, let it do
the indexing on its own (no need for a next pointer,) and off you go.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@aja.com
Sent: Tuesday, September 11, 2001 8:23 PM
To: NT Developers Interest List
Subject: [ntdev] Scatter-Gather list across PCI bridge

Hello, brains. :-}
I’m trying to implement Scatter/Gather DMA on a PCI device,
where the
chaining list resides in Host memory on the PC. I’m using
ExAllocatePool
to allocate a chunk of memory where I will build the list. I
write the
Host Address, Device Address, Length, and Next Descriptor
Link (always this
one +1) for each descriptor. My hardware guy tells me that
the address of
the list (and the NextDescriptor fields) is not a valid PCI
address, as
seen from the device. (The Host Address does seem to be valid.)
Perhaps I should be using a different method to allocate
the memory for
the list (I’m not really familiar with HalTranslateBusAddress
and such, so
don’t know how that fits in…). Or maybe I need to do
something extra to
the address I get from ExAllocatePool?
I’m confused. Can anyone spin me around and head me in the right
direction?
Dave Taylor
AJA Video Systems
xxxxx@aja.com


You are currently subscribed to ntdev as:
xxxxx@hollistech.com To unsubscribe send a blank email to
leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: xxxxx@compuware.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Thanks for the clear explanation, Mark. Isn’t it amzing how a newbie can
read every book on a subject and still have no cue when to apply it in the
real world? :slight_smile:
Dave Taylor
AJA Video Systems
xxxxx@aja.com

Mark Roddy wrote:

Hmmm…

Hard to know where to start with this other than to agree that you are
confused :slight_smile:

First, lets review the difference between Physical Address and Virtual
Address. Software running on the NT platform only uses virtual addresses
for memory. Memory management hardware and software translate virtual
addresses to physical addresses in order to locate the physical memory
used for any particular virtual address. The mapping from virtual to
physical can change over time.

PCI devices accessing host memory only use Physical Addresses. If you
hand a PCI device a virtual address and instruct it to perfrom some
transfer operation in or out of that address, it is very unlikely that
your request will be performed as you intended. This is why your
‘hardware guy’ was complaining about invalid PCI addresses.

When you call ExAllocatePool you obtain a virtual address of a buffer.
That virtual address maps to some number of physical page addresses.
There is no guarantee that the physical pages are contiguous (although
if you specified nonpagedpool in general they will be contiguous.) But,
nevermind that, you need a physically contiguous buffer, and you need to
use physical, not virtual, addresses for everything that your PCI device
uses.

The correct API is AllocateCommonBuffer (or if we are talking about NT4
rather than W2K HalAllocateCommonBuffer.) This API returns to you BOTH a
virtual address and a physical address (actually the documentation
refers to it as a logical address, but lets not get into that right
now.) Better yet, the api provides a guarantee that the buffer returned
is physically contiguous.

Now you can use this buffer to construct your SGL, typically as a count
plus a variable length array of {physical address, length} elements.
Inform your PCI device of the physical address of your SGL, let it do
the indexing on its own (no need for a next pointer,) and off you go.

> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@aja.com
> Sent: Tuesday, September 11, 2001 8:23 PM
> To: NT Developers Interest List
> Subject: [ntdev] Scatter-Gather list across PCI bridge
>
>
> Hello, brains. :-}
> I’m trying to implement Scatter/Gather DMA on a PCI device,
> where the
> chaining list resides in Host memory on the PC. I’m using
> ExAllocatePool
> to allocate a chunk of memory where I will build the list. I
> write the
> Host Address, Device Address, Length, and Next Descriptor
> Link (always this
> one +1) for each descriptor. My hardware guy tells me that
> the address of
> the list (and the NextDescriptor fields) is not a valid PCI
> address, as
> seen from the device. (The Host Address does seem to be valid.)
> Perhaps I should be using a different method to allocate
> the memory for
> the list (I’m not really familiar with HalTranslateBusAddress
> and such, so
> don’t know how that fits in…). Or maybe I need to do
> something extra to
> the address I get from ExAllocatePool?
> I’m confused. Can anyone spin me around and head me in the right
> direction?
> Dave Taylor
> AJA Video Systems
> xxxxx@aja.com
>
> —
> You are currently subscribed to ntdev as:
> xxxxx@hollistech.com To unsubscribe send a blank email to
> leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com
>
>


You are currently subscribed to ntdev as: xxxxx@aja.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

> Hello, brains. :-}

I’m trying to implement Scatter/Gather DMA on a PCI device, where the
chaining list resides in Host memory on the PC. I’m using ExAllocatePool
to allocate a chunk of memory where I will build the list. I write the

The steps are:

  1. use HalAllocateCommonBuffer for it instead.
  2. Use it once for a large buffer at device init time only.
  3. Do your own sub-allocation (list-based) of this buffer when you build the
    list.

Max


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

> Hi Mark,

I got a small about HalallocateCommonBuffer.
Will the physical address is contigouus…

Yes, surely.

Max


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com