HI,
I’m new in this forum…
I read this interesting post http://www.osronline.com/showThread.CFM?link=138520
Consider for instance a PC with a PCI bus. My question is: PCI bus-relative physical addresses are involved when either a PCI device bus-master (e.g NIC) or Memory controller (North bridge) assert an address on PCI bus to read/write a location that exist in PCI device internal register or in main system RAM ?
thanks
xxxxx@alice.it wrote:
Consider for instance a PC with a PCI bus. My question is: PCI bus-relative physical addresses are involved when either a PCI device bus-master (e.g NIC) or Memory controller (North bridge) assert an address on PCI bus to read/write a location that exist in PCI device internal register or in main system RAM ?
I didn’t quite understand your question, but that’s not going to stop me
from writing an answer.
Consider this sophisticated ASCII art picture:
|
|
CPU -> NBridge -> PCI Controller -> PCI Devices
|
|
If the address space on the right side of that line (that is, the PCI
bus view) is different from the address space on the left side of that
line (that is, the CPU view), then the PCI controller will ordinarily do
any rewriting that is necessary. So, if your driver refers to registers
on your board, the driver uses CPU-addressing, and the PCI controller
will automatically do the address translation. You don’t have to worry
about it.
The only time this process has to become visible is when a PCI device
needs to become bus master. In that case, your driver, which is
accustomed to working with CPU-relative addresses, needs to hand an
address to the device, which must work with bus-relative addresses. So,
Windows will tell you what bus-relative address your board must use in
order to access the memory you need.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
Thanks for answer…
you have drawn dividing line in the middle of PCI controller…so, from a logical point of view, NBridge knows only CPU-relative physical addresses and redirect them toward system RAM or PCI controller (that in turn could perform address translation)…is it right ?
another point: driver code is accustomed to work with virtual CPU addresses or CPU-relative physical addresses ?
xxxxx@alice.it wrote:
you have drawn dividing line in the middle of PCI controller…so, from a logical point of view, NBridge knows only CPU-relative physical addresses and redirect them toward system RAM or PCI controller (that in turn could perform address translation)…is it right ?
Yes.
another point: driver code is accustomed to work with virtual CPU addresses or CPU-relative physical addresses ?
Well, I’m not sure that’s really the right way to think about it.
Almost all of the time, you work with virtual addresses, and you don’t
think about these kinds of issues. It’s only when you have to deal with
hardware that these kinds of things come up.
CPU-relative physical addresses don’t arise all that often. The
“translated resources” that you get in your AddDevice handler are CPU
physical addresses, but all you do is hand them to MmMapIoSpace to
convert them to virtual. When you need a physical address, it’s because
you need it to give to some piece of bus-mastering hardware, where what
you really want is bus-relative addresses.
Now, this discussion is a little bit catty, because 99.9% of the time,
CPU-relative physical addresses and bus-relative physical addresses are
exactly the same. But it’s important to think about things in the right
context for those times it DOES come up.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
Which tool can i use on Win XP system (32 bit) to check how system RAM page frames are mapped in CPU-relative phy address space ? AFAIK device manager show only I/O ports and memory-mapped device internal registers…
xxxxx@alice.it wrote:
Which tool can i use on Win XP system (32 bit) to check how system RAM page frames are mapped in CPU-relative phy address space ? AFAIK device manager show only I/O ports and memory-mapped device internal registers…
RAM is, by definition, always in CPU physical space.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
The way they are mapped is dependent on the contents of the current page
map. The CPU presents virtual addresses to its page-mapping logic, and
the page-mapping logic uses the contents of the page table that is
currently in effect; this mapping is accelerated by the use of tricks like
the “TLB” (Translation Lookaside Buffer" which is a cache of the most
recently translated virtual addresses. This mapping will change moment to
moment, such as every time a page fault occurs; in fact, the page table of
process X might be modified by a page fault in process Y, because some
page in X is the least-recently-used page and consequently will get paged
out to make room for the page needed by process Y.
So if you happen to discover the virtual-physical mapping, unless you have
the kernel stopped in a kernel debugger and do not allow any execution to
proceed, then you have a snapshot of the current mapping. But let the
kernel run for a few milliseconds, and your information can be
out-of-date. And it is not at all clear what the usefulness of such
knowledge would be.
I suggest reading the Intel manual about how paging tables work.
Particularly Chapter 3 of Volume 1 (at least that’s where it is in my
downloaded copy). Also look at Volume 3A, the section called “Protected
Mode Memory Management”
joe
Which tool can i use on Win XP system (32 bit) to check how system RAM
page frames are mapped in CPU-relative phy address space ? AFAIK device
manager show only I/O ports and memory-mapped device internal
registers…
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
Depends on how you interpret “physical space”. If you mean the memory
address lines coming out of the chip, then yes, RAM is always in physical
address space. But that is a pretty useless piece of knowledge, because
the CPU, while executing instructions, can only present virtual addresses
(at least when running Windows), and the precise mapping of these virtual
addresses to physical addresses is defined by the current page map in
effect. And therefore can change several hundred times per second.
joe
xxxxx@alice.it wrote:
> Which tool can i use on Win XP system (32 bit) to check how system RAM
> page frames are mapped in CPU-relative phy address space ? AFAIK device
> manager show only I/O ports and memory-mapped device internal
> registers…RAM is, by definition, always in CPU physical space.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
xxxxx@flounder.com wrote:
Depends on how you interpret “physical space”. If you mean the memory
address lines coming out of the chip, then yes, RAM is always in physical
address space.
I believe I was quite explicit in stating my interpretation of “physical
space”. This has been a discussion of map registers, and the
distinction between CPU physical space and bus physical space. My key
point was that RAM lives on the CPU side of any bus interfaces, so RAM
addresses are in the CPU’s view of physical space, not any bus remapping.
But that is a pretty useless piece of knowledge, because
the CPU, while executing instructions, can only present virtual addresses
(at least when running Windows), and the precise mapping of these virtual
addresses to physical addresses is defined by the current page map in
effect. And therefore can change several hundred times per second.
You’re correct that it is pretty useless, but that’s irrelevant to the
conversation. We’re talking about the physical addresses coming out of
the CPU’s pins.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
Yes, Tim is correct. My question was related to physical addresses coming out of the CPU’s pins in ordert to better understand PC systems
coming back to my question…I mean a way (a tool or other…) to find out in which CPU physical address ranges RAM page frames are mapped to
Thanks
> I mean a way (a tool or other…) to find out in which CPU physical address ranges RAM page
frames are mapped to
Well, I don’t know about Windows, but under Linux you can simply type cat /proc/iomem. For example, look what physical address space layout at my laptop is like:
00000000-0000ffff : reserved
00010000-0009c3ff : System RAM
0009c400-0009ffff : reserved
000dc000-000dffff : reserved
000e4000-000fffff : reserved
00100000-b327bfff : System RAM
01000000-012b38ec : Kernel code
012b38ed-01474baf : Kernel data
01512000-015d8973 : Kernel bss
b327c000-b3281fff : reserved
b3282000-b33e1fff : System RAM
b33e2000-b340efff : reserved
b340f000-b346efff : System RAM
b346f000-b34effff : ACPI Non-volatile Storage
b34f0000-b370efff : reserved
b370f000-b3716fff : System RAM
b3717000-b371efff : reserved
b371f000-b377cfff : System RAM
b377d000-b379efff : ACPI Non-volatile Storage
b379f000-b37e0fff : System RAM
b37e1000-b37fefff : ACPI Tables
b37ff000-b37fffff : System RAM
b3800000-bfffffff : reserved
c0000000-c01fffff : PCI Bus 0000:04
c0200000-c0200fff : Intel Flush Page
d0000000-dfffffff : 0000:00:02.0
e0000000-efffffff : PCI MMCONFIG 0000 [bus 00-ff]
e0000000-efffffff : reserved
e0000000-efffffff : pnp 00:09
f0000000-f03fffff : 0000:00:02.0
f0400000-f04fffff : PCI Bus 0000:02
f0400000-f040ffff : 0000:02:00.0
f0400000-f040ffff : tg3
f0500000-f05fffff : PCI Bus 0000:04
f0500000-f050ffff : 0000:04:00.0
f0500000-f050ffff : ath9k
f0800000-f0803fff : 0000:00:1b.0
f0800000-f0803fff : ICH HD audio
f0804000-f080400f : 0000:00:16.0
f0806000-f08063ff : 0000:00:1a.0
f0806000-f08063ff : ehci_hcd
f0807000-f08073ff : 0000:00:1d.0
f0807000-f08073ff : ehci_hcd
f0808000-f08087ff : 0000:00:1f.2
f0808000-f08087ff : ahci
f0809000-f08090ff : 0000:00:1f.3
f080a000-f080afff : reserved
f080a000-f080afff : 0000:00:1f.6
f0a00000-f0bfffff : PCI Bus 0000:02
feaff000-feafffff : reserved
feaff000-feafffff : pnp 00:09
fec00000-fec0ffff : reserved
fec00000-fec003ff : IOAPIC 0
fed00000-fed003ff : HPET 0
fed00000-fed003ff : reserved
fed10000-fed13fff : pnp 00:09
fed18000-fed18fff : pnp 00:09
fed19000-fed19fff : pnp 00:09
fed1c000-fed8ffff : reserved
fed1c000-fed1ffff : pnp 00:09
fed20000-fed3ffff : pnp 00:09
fed40000-fed44fff : pnp 00:09
fed45000-fed8ffff : pnp 00:09
fee00000-fee00fff : Local APIC
fee00000-fee00fff : reserved
ff000000-ffffffff : reserved
ff800000-ff800fff : pnp 00:05
100000000-137ffffff : System RAM
As you can see, this output combines the results of 0xe820 map obtained by legacy 0x15 calls with the ones obtained from ACPI tables…
Anton Bassov
great !
IIUC PCI Bus 0000:02 and PCI Bus 0000:04 identify two PCI buses in your laptop…
Addresses shown are in CPU-relative space and I suppose there is no adress translation to/from PCI bus-relative address space…For instance on PCI bus 02 exist a device/function 02:00.0 (tg3) - CPU accesses this device’s internal memory (registers) asserting on its pin addresses in the range f0400000-f040ffff…
I guess there is system RAM mapped above 4GB boundary, right ?
xxxxx@alice.it wrote:
Addresses shown are in CPU-relative space and I suppose there is no adress translation to/from PCI bus-relative address space…
Probably not, but this table wouldn’t show you that anyway. This is a
table of CPU-relative physical addresses.
For instance on PCI bus 02 exist a device/function 02:00.0 (tg3) - CPU accesses this device’s internal memory (registers) asserting on its pin addresses in the range f0400000-f040ffff…
I guess there is system RAM mapped above 4GB boundary, right ?
Yes. There always is in a system with 4GB of RAM.
…
ff000000-ffffffff : reserved
ff800000-ff800fff : pnp 00:05
100000000-137ffffff : System RAM <<<
The BIOS has to carve out a section of physical address space below 4GB
in which it can locate PCI devices. On this system, that hole is the
gigabyte starting at C0000000. The memory that would have been located
there is instead relocated above the 4GB boundary. The 32-bit client
systems (XP, Vista, Win 7) cannot access that memory – it is wasted.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
xxxxx@alice.it wrote:
coming back to my question…I mean a way (a tool or other…) to find out in which CPU physical address ranges RAM page frames are mapped to
There are tools that can show you that, but there are no APIs, because
that information is just not useful.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
Good…consider now Windows 32-bit client running on this PC system. It mantein a PFN database with entries for each physical page memory
Based on this physical space layout a question arise: has PFN database entries also for memory-mapped physical ranges (e.g. memory-mapped PCI device’s registers addresses) ? I guess will exist CPU Virtual Address (VAs) these physical ranges are mapped to…
xxxxx@alice.it wrote:
Good…consider now Windows 32-bit client running on this PC system. It mantein a PFN database with entries for each physical page memory
Based on this physical space layout a question arise: has PFN database entries also for memory-mapped physical ranges (e.g. memory-mapped PCI device’s registers addresses) ? I guess will exist CPU Virtual Address (VAs) these physical ranges are mapped to…
Who cares? That’s a Windows kernel implementation detail that doesn’t
matter to either drivers or applications. Windows sets up the page
tables. We don’t care how.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
As far I read Windows Internals and I was just curious about this topic
Anyway IIRC when driver’s ISR code is running (in interrupt context) it shouldn’t access controlled device’s registers via VAs because it could incurr in a BSOD (page tables could be paget out) So i guess ISR should use always CPU-relative physical addresses
Does it make sense ? Thanks
When CPU is in protected mode it can access memory location only by a virtual address that it translates into a physical one (which is CPU-relative by its very definition). It does not necessarily mean that this location actually refers to RAM - it can be ROM, memory-mapped device, APIC, etc. However, it has to be accessed only as a virtual address…
It is OS’s responsibility to ensure that accessing memory that had been mapped with MmMapIoSpace() does not result in page faults. Therefore, all page tables that are used for mapping the target page are going to be locked in RAM…
Anton Bassov
This would make no sense, because for every real device it is ESSENTIAL
that the ISR be able to read or write the control registers for the
device. Since the page tables that hold this mapping treat virtual
addresses on devices as if they are honorary members of the non-paged
pool, the kernel maps would never be paged out.
There is not a single instance of code in existence in Windows, nor has
there EVER been, that lets the processor access physical memory from an
instruction. The code stream can ONLY issue virtual (32 or 64 bit)
addresses. The only way you can access addresses that are on the bus is
if there is a page table entry that provides this mapping. And the page
tables to the nonpaged pool, for obvious reasons, are not paged.
When you get a BAR address from a PCI card, it is the bus-relative address
that has been assigned by the PCI BIOS and friends. Before you can axcess
it, you must call HalTranslateBusAddress, and you get back a virtual
address that will map to the correct physical address coming out of the
CPU so that ultimately the right device will respond to that signal. In
modern drivers, this is done for you by the PNP system, and this is
presented to you, the driver writer, as the “translated” resources in the
IRP_MJ_PNP:IRP_MN_START_DEVICE notifivation, and is handled in the KMDF
framework as partof the “notification” handling, so you never see the
antique legacy HAL calls in a PnP or KMDF-based driver.
The two most important registers which you MUST access for a PCI device is
a status indication somewhere that says “I have asserted interrupt” and
whatever register values must be manipulated to tell the device, “Thank
you very much, I am handling the interrupt, so please deassert the
interrupt signal”. You simply can’t write an ISR for a PCI device without
accessing and manipulating this state.
Note that since it is IMPOSSIBLE for instructions that are executing with
Virtual Memory enabled to create a bus-relative physical address, you can
ONLY use virtual addresses.
Some years ago I taught a one-shot course in kernel debugging. This was a
weird experience; the students gave ME homework! On Monday, they handed
me a crash dump and said “Tell us what went wrong”. I was given no other
detail. It was taking a BSOD on a MOV instruction that was reading a
device register. There were lots of possible causes, but I was asking
questions during my presentation of the analysis, and discovered that this
particular device was the disk controller driver, and about every 3 months
the server would die. I told them it was a hardware problem, quite likely
caused by an out-of-spec device on the bus screwing up the bus signals.
Turns out not one of these driver guys had a clue as to how a PCI bus
actually works! So I improvised a three-hour lecture on the PCI bus
architecture, pointing out that it was not necessarily their device that
was defective; it might have been the victim of collateral damage from a
drive-by waveform distortion. A device with bad bus protocol down to the
level of small-number-of-nanoseconds timing skews was probably the
problem. I suggested ideas like remove any cards that are not essential
and if te problem goes away, it might be one of those cards; if the
problem stays, it might be your card. Or not. I gave them a 20-minute
nanocourse on analog signals, impedances, reflected signals, capacitative
and inductive bus loading, etc. It was all new to them. I even mentioned
ground loops (in one infamous incident, a colleague was told the driver
was defective because it would reprogram the color maps in the display
with apparently random values. Using a PCI bus analyzer and many days of
analysis, he showed that at the time the screen went weird, there were no
bus cycles addressing the palette registers. Turns out te problem was a
ground loop under the video chip. I told them several other stories along
this line, such as the tape drive controller whose status register would
quite often deliver random bits. We knew this because the value had many
of the “unused, always returns 0” bits set to 1s. The programmer modified
the driver so that it had to get three successive identical values before
it would trust the status bits. The overnight backups stopped failing
with “unrecoverable tape drive error”. But the guys I was lecturing
thought that new hardware always worked perfectly. So I told them about
my uncle, who had been in the Nigerian army, and had $30,000,00 in gold
stashed away and I needed their help to get it out of the country. I
figured, if these guys believe new hardware is perfect, they’ll believe
anything!)
But the key part of the analysis was that you will NOT get an invalid
address when you access a device unless you have accidentally clobbered
the page table entry for that mapping or the device is determined to be
non-responsive. The first was unlikely because the address I was seeing
was in valid device space, a register located immediately before it had
been successfully read about 200 instructions earlier, and the problem
happened very rarely for a device that was handling hundreds of bus-level
transactions per second. I led them through a complete failure-tree
analysis and explained why I was reasonably confident about ruling out
most of the branches.
joe
As far I read Windows Internals and I was just curious about this topic
Anyway IIRC when driver’s ISR code is running (in interrupt context) it
shouldn’t access controlled device’s registers via VAs because it could
incurr in a BSOD (page tables could be paget out) So i guess ISR should
use always CPU-relative physical addressesDoes it make sense ? Thanks
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
I am not aware that any such “database” actually exists. For one thing,
it would duplicate information already present in the page maps, and
therefore the same information can be derived,
If such a map existed, it would have to manage the information about
device memory (whether device-owned memory or device registers which are
memory-mapped). But I am not aware any such table exists. When you (Or
theframework, these days) call HalTranslateBusAddress, they present a
64-bit physical address. What comes back is a 32-bit or 64-bit virtual
address the instruction stream can use to generate references to these
locations, and it has updated the kernel memory map to translate this
istruction-stream virtual address to a physical address. That’s all. I
have no idea if therevis any other mapping managed by this logic, but
having worked in the low-level guts of several operating systems in my
life, I never once saw anything resembling the table you propose.
And while it is useful knowledge to satisfy personal curiosity (“how does
that really work?”), it actually has zero impact on driver writers. We
get virtual addresses, we use them, and they work. Whether it is done
with auxialiary tables or smoke-and-mirrors is generally of no interest to
the driver writer. That’s why we work with the abstraction with no
serious concerns about the implementation details.
joe
Good…consider now Windows 32-bit client running on this PC system. It
mantein a PFN database with entries for each physical page memoryBased on this physical space layout a question arise: has PFN database
entries also for memory-mapped physical ranges (e.g. memory-mapped PCI
device’s registers addresses) ? I guess will exist CPU Virtual Address
(VAs) these physical ranges are mapped to…
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer