“xxxxx@flounder.com” wrote in message
news:xxxxx@ntdev:
> No, a map register is used to map 32-bit DMA to physical addresses on a
> wider bus. It is not “located” anywhere except in the I/O controller
> (usually in the bus interface chip, commonly referred to as the “North
> Bridge” but is officially the “Hub Control Interface” (HCI) chip.
> ****
Actually the above is not totally correct. The one time I encountered
hardware map registers in Windows they were used for scatter gather
handling not for mapping a smaller address to a wider address. In the
case of the SGI systems that had them everything was 32-bits.
Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr
workingmailing@163.com wrote:
Thank you Peter, what you mean is this map register is something located lower than 4G memory, if that do not exist when no needed, so OS could say – lie, that it is unlimited.
But when it is needed, such as 8G system, so OS tell you the true, not so much bounce buffer I could allocated for you, it is in fact it is a warming.
Your wording is a little confused, but I think you are starting to get
the basic idea. On a common computer system today with less than 4GB of
RAM, the CPU physical addresses always match the bus physical
addresses. No address translation is necessary. Therefore, there are
no physical “map registers”. Nothing is allocated. No memory is
required. The system could advertise that it has a billion map
registers, because they don’t really exist. They are fake.
But once you have more than 8GB of RAM, a device that can only handle
32-bit addresses needs help. Windows implements that help using bounce
buffers. The way you refer to bounce buffers is through “map
registers”. Because there are a very small number of bounce buffers,
that puts a limit on the number of “map registers” that are available.
Even with 8GB RAM, if your device can handle 64-bit addresses (as it
should), then there will be no limit on the number of map registers you
can get.
But in fact I still could not master what the reason,
As I known, map register is bounce buffer allocated by DMA abstraction or HAL or somethings else in the Windows XXX manager lower than 4G RAM phy address.
If you are trying to grasp the real details, then no, this is NOT true.
Map registers in Windows are an ABSTRACTION. They are a way to hide the
ugly details of the hardware. They refer to different things at
different times on different systems. Sometimes, map registers mean
nothing at all, like on a normal system with less than 4GB of RAM. You
are supposed to ask for them, and the system will deliver them, but they
don’t mean anything. They don’t really exist. They are just numbers.
Because of that, IN THIS CASE, the system can deliver as many as you
need. They take no resources.
On other systems, map registers refer to bounce buffers. This is the
case on a system with more than 4GB RAM with hardware that can only
handle 32-bit addresses. The number of bounce buffers is quite limited,
so IN THIS CASE the number of map registers is limited. Here, a “map
register” is just a handle that refers to a bounce buffer. There is no
such thing as a physical “map register”.
On other systems, map registers refer to real registers in hardware.
This can be the case on computers with very high end I/O systems, with
multiple buses that use different address spaces. In this case, the
number of map registers is determined by the bus hardware.
Windows created the “map register” abstraction so you would not have to
worry about these details. You follow the rules, and Windows will take
care of the hardware.
So in my thinking, map register (each is 4k/one page) is allocate by OS in the lower 4G phy memory working as bounce buffer, which will add one copy from bounce buffer to requestor buffer if the requestor buffer located in PHY memory higher than 4G.
That’s one thing it might mean. If you have more than 4GB RAM and a
device limited to 32 bit addressing, then map registers will refer to
bounce buffers.
So windows have a limit on map register numbers?
No, but Windows has a very small limit on the number of bounce buffers.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
thank you, Tim, and every one.
Even though “map register” is a abstraction, which at some percentage, driver writer need not put so much time on it, even you put time on it, you may not get very clear picture of it.
But from this discussion, and so many experts give very useful imformation, I feel I learn something and it is useful for me on driver/system develop in the future of my working.
thank you again for everybody.
> As a non-real-world example, consider a 32-bit PCI bus attached to a 64-bit machine. Say that the motherboard hardware has the ability to “relocate” that entire PCI bus, so that physical address range 1234_0000_0000 to 1234_ffff_ffff maps to that extra PCI bus. In that case, the hardware would have map registers to map the CPU’s physical address 1234_0000_0000 to that bus’s physical address 0. The driver would work with physical addresses in the 123400000000 range, but the DMA abstraction would convert that to physical address 0 for you to send to the hardware.
just a beginner’s doubt…in Tim non-real-world example PCI bus is “relocated” in CPU’s physical address range 1234_0000_0000 to 1234_ffff_ffff. In this physical range devices’s registers (memory-mapped IO) and a quota of RAM page frames are mapped to from CPU’s view
Now I guess translations in the two directions (via map registers) apply to different address ranges. I mean…in downstream direction (from CPU to PCI bus devices registers) certain subsets of range 1234_0000_0000 - 1234_ffff_ffff are translated in corresponding bus-relative addresses PCI devices registers are mapped to
In the other direction (from PCI devices perfoming DMA to main memory) translation apply to different ranges of bus-relative addresses (of course where the quota of RAM page frames are mapped to from PCI bus-relative address space point of view)
Does it make sense ? Thanks
xxxxx@alice.it wrote:
just a beginner’s doubt…in Tim non-real-world example PCI bus is “relocated” in CPU’s physical address range 1234_0000_0000 to 1234_ffff_ffff. In this physical range devices’s registers (memory-mapped IO) and a quota of RAM page frames are mapped to from CPU’s view
Not quite. The mapping of the device registers to CPU view is done with
the normal PCI address mapping. Nothing special is required in this
example. The CPU view addresses will (probably) be just below 4GB,
where PCI devices normally live.
That range from 1234_0000_0000 to 1234_ffff_ffff is not a CPU view
address range. Those are bus-view addresses that allow the device to
access physical RAM. They will never be used as physical addresses from
the CPU. Those are just data items handed to the driver so it can write
them to the device for doing DMA.
Now I guess translations in the two directions (via map registers) apply to different address ranges. I mean…in downstream direction (from CPU to PCI bus devices registers) certain subsets of range 1234_0000_0000 - 1234_ffff_ffff are translated in corresponding bus-relative addresses PCI devices registers are mapped to
No. The downstream direction is handled exactly like with any normal
device. The PCI controller has a physical address range (in CPU view)
that it responds to, which includes the addresses of the PCI devices
beyond it.
In the other direction (from PCI devices perfoming DMA to main memory) translation apply to different ranges of bus-relative addresses (of course where the quota of RAM page frames are mapped to from PCI bus-relative address space point of view)
PCI devices performing DMA will use those 1234_0000_0000 addresses. The
PCI controller in the upstream direction will convert those to CPU view
addresses in order to get to RAM.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
> It is not “located” anywhere except in the I/O controller (usually in the bus interface chip,
commonly referred to as the “North Bridge” but is officially the “Hub Control Interface” (HCI) chip.
Actually, the I/O controller is referred to as the “South Bridge”. The term “North Bridge” refers to (fast) memory controller that, unlike “South Bridge”(slow one), is directly connected to the CPU. The current trend seems to be integration of memory controller into CPU die itself, effectively eliminating the “North Bridge” chip. “South Bridge” is directly connected to the CPU on such a system
Furthermore, the term HCI (Host controller interface) normally refers to USB, Firewire and SATA controllers. If you mean “South Bridge”, the official term for it is ICH (IO Controller Hub). The official term for the “North Bridge” is Memory Controller Hub (MCH)…
Anton Bassov
> That range from 1234_0000_0000 to 1234_ffff_ffff is not a CPU view address range. Those are bus-view addresses that allow the device to access physical RAM. They will never be used as physical addresses from the CPU. Those are just data items handed to the driver so it can write them to the device for doing DMA.
PCI devices performing DMA will use those 1234_0000_0000 addresses. The PCI controller in the upstream direction will convert those to CPU view addresses in order to get to RAM
Tim, maybe I’m wrong but in final part of your previous post (13 Apr 12):
In that case, the hardware would have map registers to map the CPU’s physical address 1234_0000_0000 to that bus’s physical address 0. The driver would work with physical addresses in the 123400000000 range, but the DMA abstraction would convert that to physical address 0 for you to send to the hardware
I understand that DMA bus-master’s devices have to use PCI bus-relative address 0 in order to access CPU’s physical address 1234_0000_0000 (where a RAM page frame is mapped to). It is in charge of map register (hardware IOMMU in the example) convert PCI target address 0 (asserted by DMA bus master’s device) to physical address 1234_0000_0000
xxxxx@alice.it wrote:
Tim, maybe I’m wrong but in final part of your previous post (13 Apr 12):
> In that case, the hardware would have map registers to map the CPU’s physical address 1234_0000_0000 to that bus’s physical address 0. The driver would work with physical addresses in the 123400000000 range, but the DMA abstraction would convert that to physical address 0 for you to send to the hardware
I understand that DMA bus-master’s devices have to use PCI bus-relative address 0 in order to access CPU’s physical address 1234_0000_0000 (where a RAM page frame is mapped to). It is in charge of map register (hardware IOMMU in the example) convert PCI target address 0 (asserted by DMA bus master’s device) to physical address 1234_0000_0000
I think my paragraph was written a little too quickly, or without enough
attention to detail. The “map registers” are a mapping from bus space
to CPU space, so that the device can access real memory (which is always
in CPU space).
Say we have a machine with 2GB of RAM. Say that machine has a PCI
controller that claims 64MB of address space, and say there is a device
on that bus that needs a 64KB BAR. Let’s say that PCI controller gets
its share at E000_0000, and let’s say that the device’s BAR gets
assigned at offset 20000.
So, in the CPU physical view:
0000_0000 => RAM
…
7FFF_FFFF => RAM
8000_0000 => unassigned
E000_0000 => PCI controller
E002_0000 => Our PCI device BAR
Now, let’s say the PCI controller has to use map registers to get back
to physical RAM. We want to do DMA to physical address 6000_0000 in our
real RAM. The PCI driver create map registers in the PCI controller.
Let’s say it picks the address 1234_6000_0000 for that mapping.
Now, the PCI bus view of physical space is:
0000_0002_0000 => Our PCI device BAR
1234_6000_0000 => physical RAM, at 6000_0000 in the CPU view
So, my driver has to create a DMA request by writing to my device
registers (which are at E002_0000 + x), writing the physical address
1234_6000_0000. When I trigger the DMA request, my device reads from
1234_6000_0000. The PCI controller, thanks to the map registers, maps
that to 6000_0000, and reads from that address in RAM.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
Good…but IIRC we consider in the example a 32-bit PCI bus…how can the PCI device read from 1234_6000_0000 when PCI bus address space is 32 bit widht ?
xxxxx@alice.it wrote:
Good…but IIRC we consider in the example a 32-bit PCI bus…how can the PCI device read from 1234_6000_0000 when PCI bus address space is 32 bit widht ?
It could not be a 32-bit bus.
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
Just to clarify the obscure naming convention: back in the early 1990s,
when I first heard these terms, my basic reaction was “WTF” (that
abbreviation, as far as I know, came many years later). The hardware guy
said, “In box diagrams that show the components, the memory controller
always appears above the PCI bus in the drawing, and the bus interface
(mostly ISA in those days) appears below the PCI bus in the illustration.”
Hence the names “north” and “south” became popular usage. I used these
terms when I taught a course at Intel in the early 2000s, and was roundly
scolded by the engineers there, who informed me that such terms were
obsolete.
Also, make sure you are not talking to an Australian engineer. Google
“australian upside down map”.
I remember in the late 1980s, when the EISA bus was still a serious
contender for the standard bus architecture, having a discussion with a
German engineer about the ee-sa bus and the eye-sa bus. By “eye-sa”, an
American thinks “ISA” and for “ee-sa” an American thinks “EISA”. I
finally realized our communication breakdown was due to the fact that a
German reads “ISA” as “ee-sa” and “EISA” as “eye-sa”
joe
> It is not “located” anywhere except in the I/O controller (usually in
> the bus interface chip,
> commonly referred to as the “North Bridge” but is officially the “Hub
> Control Interface” (HCI) chip.Actually, the I/O controller is referred to as the “South Bridge”. The
term “North Bridge” refers to (fast) memory controller that, unlike “South
Bridge”(slow one), is directly connected to the CPU. The current trend
seems to be integration of memory controller into CPU die itself,
effectively eliminating the “North Bridge” chip. “South Bridge” is
directly connected to the CPU on such a systemFurthermore, the term HCI (Host controller interface) normally refers to
USB, Firewire and SATA controllers. If you mean “South Bridge”, the
official term for it is ICH (IO Controller Hub). The official term for the
“North Bridge” is Memory Controller Hub (MCH)…Anton Bassov
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
The bus on high-end 32-bit systems is typically a 36-bit bus, allowing for
64GB of physical memory. While a running 32-bit program in Win32 cannot
access more than 2GB/3GB, the page tables translate the 32-bit virtual
addresses to 36-bit memory-bus addresses. The purpose of a map register
is to allow a 32-bit-address PCI card to do DMA directly into memory above
the 4GB boundary by mapping the adress put on the PCI bus by the bus
master into the larger address space.
My recollection is that the AMD64 chips implement a 40-bit bus address
coming out of the CPU, so a 32-bit process running under WoW64 could have
every one of its 32-bit virtual addresses mapped to physical memory > 4GB.
Note that 40 bits of address allow for 1TB of physical memory.
joe
xxxxx@alice.it wrote:
> Good…but IIRC we consider in the example a 32-bit PCI bus…how can
> the PCI device read from 1234_6000_0000 when PCI bus address space is 32
> bit widht ?It could not be a 32-bit bus.
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
> Also, make sure you are not talking to an Australian engineer. Google
“australian upside down map”.
So “The stock went south” is actually a good thing in Australia?
Calvin
Somebody else probably already said this, but I want to say it succinctly.
PCI buses are device in the PCI spec as having a 64-bit memory address
space. Period.
You can legally build a device that can only do 32-bit decode on that bus
(by expressing its BARs as 32-bits wide) or you can build a device with some
limit on the addresses that are available in your on-device DMA engine.
But the bus definition itself is always 64-bits wide.
With that said, many implementations will throw and error if you do DMA
above the physical address width of the processor, typically about 44 bits
at the moment.
Jake Oshins
Windows Kernel Team
This message implies no warranties and confers no rights.
wrote in message news:xxxxx@ntdev…
Good…but IIRC we consider in the example a 32-bit PCI bus…how can the
PCI device read from 1234_6000_0000 when PCI bus address space is 32 bit
widht ?
Back in the old PCI days, DMA engine with 32-bit address line can access
memory beyond 4G by asserting DAC (dual address cycle) as master.
Calvin
On Mon, Apr 30, 2012 at 2:57 PM, wrote:
> Good…but IIRC we consider in the example a 32-bit PCI bus…how can the
> PCI device read from 1234_6000_0000 when PCI bus address space is 32 bit
> widht ?
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
So, trying to recap…
The goal of map register (implemented here via hw IOMMU) is to perform translation ONLY in upstream direction in order to convert a bus-relative target address (read/write target of transactions started by DMA capable devices) in a CPU’s physical address where RAM location are mapped to, right ?
In the opposite direction (CPU access to devices’s registers) there is NO translations but just normal behaviour of PCI controller windows…
xxxxx@alice.it wrote:
The goal of map register (implemented here via hw IOMMU) is to perform translation ONLY in upstream direction in order to convert a bus-relative target address (read/write target of transactions started by DMA capable devices) in a CPU’s physical address where RAM location are mapped to, right ?
In the opposite direction (CPU access to devices’s registers) there is NO translations but just normal behaviour of PCI controller windows…
Yes.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
Thanks all experts !!
Carlo.
Well, there are no CPU->device translations in statistically very close to
100% of the machines that you use. It’s entirely possible that there could
be translations in this path. These are modeled in Windows by handing you a
set of “raw” (bus-relative) resources in your start IRP and a set of
“translated” resources in your start IRP. The translated base address is
the one that the CPU should use as its physical address – i.e. the one that
you use to call MmMapIoSpace, which is the function that creates virtual
addresses for your device’s memory.
Jake Oshins
Windows Kernel Team
This message implies no warranties and confers no rights.
“Tim Roberts” wrote in message news:xxxxx@ntdev…
xxxxx@alice.it wrote:
The goal of map register (implemented here via hw IOMMU) is to perform
translation ONLY in upstream direction in order to convert a bus-relative
target address (read/write target of transactions started by DMA capable
devices) in a CPU’s physical address where RAM location are mapped to,
right ?In the opposite direction (CPU access to devices’s registers) there is NO
translations but just normal behaviour of PCI controller windows…
Yes.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
Some years ago, Digital Equipment Corporation designed the peripherals for
their VAX systems to use virtual addresses. You programmeed the DMA
device with the virtual address. Then they came out with the Alpha
workstation, and wanted to use the same peripherals, but the page table
layouts between the VAX and the Alpha were wildly different. It may have
been the only machine that used virtual addresses for DMA, and after the
experience of trying to retrofit all the peripherals to the Alpha, they
relized that physical addresses were the only thing that made sense, and
there was a good reason everyone else in history had used physical
addresses. Then engineers who did this said, “Never again!” would they
consider using virtual addresses for DMA.
Well, there are no CPU->device translations in statistically very close to
100% of the machines that you use. It’s entirely possible that there
could
be translations in this path. These are modeled in Windows by handing you
a
set of “raw” (bus-relative) resources in your start IRP and a set of
“translated” resources in your start IRP. The translated base address is
the one that the CPU should use as its physical address – i.e. the one
that
you use to call MmMapIoSpace, which is the function that creates virtual
addresses for your device’s memory.Jake Oshins
Windows Kernel TeamThis message implies no warranties and confers no rights.
“Tim Roberts” wrote in message news:xxxxx@ntdev…
xxxxx@alice.it wrote:
> The goal of map register (implemented here via hw IOMMU) is to perform
> translation ONLY in upstream direction in order to convert a
> bus-relative
> target address (read/write target of transactions started by DMA
> capable
> devices) in a CPU’s physical address where RAM location are mapped to,
> right ?
>
> In the opposite direction (CPU access to devices’s registers) there is
> NO
> translations but just normal behaviour of PCI controller windows…Yes.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer