BUS_INTERFACE_STANDARD on a custom bus type and playing games with bus-centric physical addresses...

NTDEV
1

I have my Xen Bus (different to xenbus) which is enumerated by my xenpci
driver. It has it’s own GUID etc.

I am testing an idea of implementing BUS_INTERFACE_STANDARD and have a
couple of questions…

Basically, from the docs, I believe that the PHYSICAL_ADDRESS returned
by any of the mapping functions in the DMA_ADAPTER is supposed to be bus
relative, and does not (necessarily) correspond to the actual physical
address of the mapped buffer (eg a PCI device may have a different view
of physical memory than the CPU).

Pages of memory are passed between the virtual machine and Xen via
something called a ‘grant ref’ which is just an index into a virtual
machine allocated table. So I pass the grant ref to Xen, and it looks up
in my table to see what access is allowed to a page of memory.

Could I treat that grant ref as the physical address of a buffer in my
DMA_ADAPTER? So if something wants the physical address, it actually
gets a grant ref and that ref can just be put onto the communications
ring. This would push a lot of the complexity back onto the bus driver /
pdo’s, rather than being reimplemented into the function drivers. The
grant ref is really the bus-relative address…

I can see nothing in the docs that would make what I am trying to do
invalid, but past experience says that the docs don’t always tell the
whole story.

Thanks

James

2

So if I pass in a gref as a BusAddress to BusTranslateBusAddress, what do I
get back as a TranslatedAddress?
Mark Roddy

On Fri, Jan 23, 2009 at 11:19 PM, James Harper <
xxxxx@bendigoit.com.au> wrote:

I have my Xen Bus (different to xenbus) which is enumerated by my xenpci
driver. It has it’s own GUID etc.

I am testing an idea of implementing BUS_INTERFACE_STANDARD and have a
couple of questions…

Basically, from the docs, I believe that the PHYSICAL_ADDRESS returned
by any of the mapping functions in the DMA_ADAPTER is supposed to be bus
relative, and does not (necessarily) correspond to the actual physical
address of the mapped buffer (eg a PCI device may have a different view
of physical memory than the CPU).

Pages of memory are passed between the virtual machine and Xen via
something called a ‘grant ref’ which is just an index into a virtual
machine allocated table. So I pass the grant ref to Xen, and it looks up
in my table to see what access is allowed to a page of memory.

Could I treat that grant ref as the physical address of a buffer in my
DMA_ADAPTER? So if something wants the physical address, it actually
gets a grant ref and that ref can just be put onto the communications
ring. This would push a lot of the complexity back onto the bus driver /
pdo’s, rather than being reimplemented into the function drivers. The
grant ref is really the bus-relative address…

I can see nothing in the docs that would make what I am trying to do
invalid, but past experience says that the docs don’t always tell the
whole story.

Thanks

James

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

3

Hi James, what’s the goal you’re wanting to accomplish? Is it to move the comlexity of communicating with Xen back into the bus driver where it belongs and out of the disk and nic miniports?

4

>Basically, from the docs, I believe that the PHYSICAL_ADDRESS returned

by any of the mapping functions in the DMA_ADAPTER is supposed to be bus

PHYSICAL_ADDRESS is the “logical address”, which is the physical address passed via the mapping layer.

What is the “mapping layer” is defined by the DMA_ADAPTER implementor, in this case - by you.

So, you can give any value you want for PHYSICAL_ADDRESSes returned by your DMA adapter.

The only limitations which come to mind:

a) page granularity. If the 2 virtual addresses are in the same page - then 2 PHYSICAL_ADDRESSes must also belong to the same page.
b) common buffers must be contiguous, not only from the virtual address side (this is obvious), but from PHYSICAL_ADDRESS side too.

You can, for instance, implement AllocateCommonBuffer as ExAllocatePoolWithTag (even not MmAllocateContiguousMemory) followed by Xen grant of all these pages to consecutive Xen space.

In this case, the PHYSICAL_ADDRESSes as viewed by the client driver are actually “Xen space addresses”. The client driver can then submit these addresses to some Xen-defined facilities.

Also, in this case, MapTransfer and GetScatterGatherList should grant each of the MDL’s pages to Xen, and return these “Xen space addresses” as PHYSICAL_ADDRESSes in the SGL.

Two more important things:

a) AGP440 sample in the DDKs is a sample of DMA_ADAPTER implementation, look at it about how it works with MDLs fed to it.
b) Your “page granting” issues with NDIS are probably better solved using the DMA adapter object, not within the NDIS itself.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

5

>

So if I pass in a gref as a BusAddress to BusTranslateBusAddress, what
do
I get back as a TranslatedAddress?

As it turns out, the Logical Address has to contain the same page offset
as the Physical Address, so what I actually return is ((gref <<
PAGE_SHIFT) | offset).

So if you are asking ‘can I go backwards from gref|offset back to cpu
relative physical address’, then yes, I can get the pfn back via the
grant table to which the gref is an index. Is that what
BusTranslateBusAddress does?

James

6

You should be able to get a valid virtual address from MmMapIoSpace using
the translated physical address returned from BusTranslateBusAddress.
Mark Roddy

On Sat, Jan 24, 2009 at 6:03 PM, James Harper > wrote:

> >
> > So if I pass in a gref as a BusAddress to BusTranslateBusAddress, what
> do
> > I get back as a TranslatedAddress?
> >
>
> As it turns out, the Logical Address has to contain the same page offset
> as the Physical Address, so what I actually return is ((gref <<
> PAGE_SHIFT) | offset).
>
> So if you are asking ‘can I go backwards from gref|offset back to cpu
> relative physical address’, then yes, I can get the pfn back via the
> grant table to which the gref is an index. Is that what
> BusTranslateBusAddress does?
>
> James
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

7

>

You should be able to get a valid virtual address from MmMapIoSpace
using
the translated physical address returned from BusTranslateBusAddress.

Then yes, the right thing will happen.

The only thing that has me concerned at the moment is that the mapping
(eg MapTransfer) involves setting up the grant ref, so there needs to
exist an ‘undo’ operation that releases the grant ref. The
BuildScatterGatherList->PutScatterGatherList stuff is straightforward
and is already working for my scsiport driver, but for my NDIS driver I
actually need to fiddle with the buffers a bit (Windows and Linux have
different rules about how much header data must be on the first buffer),
so I need to copy the header into a new buffer, and possibly even
perform the IP checksum operation on it…

James

Mark Roddy

On Sat, Jan 24, 2009 at 6:03 PM, James Harper
wrote:
>
>
> >
> > So if I pass in a gref as a BusAddress to
BusTranslateBusAddress,
> what
> do
> > I get back as a TranslatedAddress?
> >
>
>
> As it turns out, the Logical Address has to contain the same
page
> offset
> as the Physical Address, so what I actually return is ((gref <<
> PAGE_SHIFT) | offset).
>
> So if you are asking ‘can I go backwards from gref|offset back
to
> cpu
> relative physical address’, then yes, I can get the pfn back via
the
> grant table to which the gref is an index. Is that what
> BusTranslateBusAddress does?
>
>
> James
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars
visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
>
> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging
and
> other seminars visit: http://www.osr.com/seminars To unsubscribe,
visit
> the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

8

>

Hi James, what’s the goal you’re wanting to accomplish? Is it to move
the
comlexity of communicating with Xen back into the bus driver where it
belongs and out of the disk and nic miniports?

Correct. Writing these drivers was the first time I even looked at the
inside of a Windows driver, and I know a whole lot more about it than I
did then.

I have already implemented this for my scsiport driver, and my
DMA_ADAPTER provides the following assistance:

. provides a bounce buffer for non-512-byte-aligned buffers
. provides a virtual address instead of a physical address for
non-read/write scsi operations

That removes a huge amount of complexity from the scsiport drivers in
exchange for a small increase in complexity in my bus driver.

I attempted writing a storport driver a while back instead of a scsiport
driver, and the big showstopper was that storport would never give me
more than 16k of buffers in a request. I think that because I had no
DMA_ADAPTER, storport was falling back to a fairly minimal mode of
operation, so I take back some of the bad things I said about it :slight_smile:

James

9

>

>Basically, from the docs, I believe that the PHYSICAL_ADDRESS
returned
>by any of the mapping functions in the DMA_ADAPTER is supposed to be
bus

PHYSICAL_ADDRESS is the “logical address”, which is the physical
address
passed via the mapping layer.

What is the “mapping layer” is defined by the DMA_ADAPTER implementor,
in
this case - by you.

So, you can give any value you want for PHYSICAL_ADDRESSes returned
by
your DMA adapter.

The only limitations which come to mind:

a) page granularity. If the 2 virtual addresses are in the same page -
then 2 PHYSICAL_ADDRESSes must also belong to the same page.

Well… that’s a fail then. Is that documented somewhere so I can read
more about it.

b) common buffers must be contiguous, not only from the virtual
address
side (this is obvious), but from PHYSICAL_ADDRESS side too.

That’s a fail too. Grant refs are simply a reference to a table, and the
table contains the PFN, and some other bits and pieces per pfn. The
grant refs are maintained on a freelist, so if I wanted to map a buffer
of pfn’s 0x6660, 0x6661, and 0x6662, they may get grant refs 16383,
16300, and 14001.

Bear in mind that Xen doesn’t care. What the virtual machine sees as
contiguous physical addresses probably actually aren’t. pfn’s 0x6660,
0x6661, and 0x6662 might in actual fact relate to mfn’s (pfn’s from
xen’s point of view) 0x1234, 0x5678, and 0xabcd. This is the reason why
you can’t just pass through a PCI device and make it work without a bit
of buffer translation.

You can, for instance, implement AllocateCommonBuffer as
ExAllocatePoolWithTag (even not MmAllocateContiguousMemory) followed
by
Xen grant of all these pages to consecutive Xen space.

In this case, the PHYSICAL_ADDRESSes as viewed by the client driver
are
actually “Xen space addresses”. The client driver can then submit
these
addresses to some Xen-defined facilities.

Also, in this case, MapTransfer and GetScatterGatherList should grant
each
of the MDL’s pages to Xen, and return these “Xen space addresses” as
PHYSICAL_ADDRESSes in the SGL.

That’s what I’ve done and it works for scsiport, which only does a
single AllocateCommonBuffer + lots of
BuildScatterGatherList+PutScatterGatherList.

Two more important things:

a) AGP440 sample in the DDKs is a sample of DMA_ADAPTER
implementation,
look at it about how it works with MDLs fed to it.

Thanks. I’ll look at that. I probably should have gone looking for a
sample earlier :slight_smile:

b) Your “page granting” issues with NDIS are probably better solved
using
the DMA adapter object, not within the NDIS itself.

That’s the plan. I actually hope to have the function device drivers not
have any awareness of grant refs at all, which would make them much
simpler, much more like hardware devices, and much more likely to make
WHQL happy.

Thanks

James

10

> Two more important things:

a) AGP440 sample in the DDKs is a sample of DMA_ADAPTER
implementation,
look at it about how it works with MDLs fed to it.

A grep for the text “DMA” does not show any matches in AGP440
(\WINDDK\6001.18002\src\kernel\agp\agp440)

Any suggestions?

James

11

What I did was create a structure that I called RING_INFO to describe and abstract the different netif and blkif structures. The ring info structure has all the information to ‘genericize’ the structures such that I can expose generic APIs from the bus like, InitializeDevice, SendMessageMdl, readResponse, etc… In the end, I don’t have to do any grant ref’ing or any other xen specific functions to communicate with the backend. All of the APIs that the bus driver exposes take a RING_INFO* as the frist parameter to the function - so when the bus gets called to put some data on the ring, it can look at the RING_INFO structure and know what ring to talk to, where in the structure the grant ref(s) are, etc - without having to have any specific knowledge of what a disk or a net is or does. As somewhat of a WDM and object purist, it appeals to me that a bus has no specific information about the devices attached to it, nor do the devices know anything about the underlying mechanisms that the bus is using to comminicate to it’s ‘hardware’. That said, some logic could be applied to the actual structure definitions to make them more friendly to our architectures - like having a common root structure and a device specific payload.

12

>address returned from BusTranslateBusAddress.

I do not think BusTranslateBusAddress is connected to DMA in any possible means. It is for the opposite direction traffic :slight_smile:

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

13

>A grep for the text “DMA” does not show any matches in AGP440

(\WINDDK\6001.18002\src\kernel\agp\agp440)

Looks like this is not the usual DMA_ADAPTER object, but something very similar wrapped in AGP_BUS_INTERFACE_STANDARD - the interface looks like the AGP-specific simpler version of the DMA adapter’s MapTransfer feature and serves the nearly same purpose.

Look at AgpMapMemory, for instance, conceptually it is very much like MapTransfer.

Look at AgpGetMappedPages, which is an amazing example of touching MDL tail directly in the WDK sample.

And so on.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com