How ACPI and PCIe are linked together

Mark_Gupta · March 28, 2012, 12:00am

Hi all,

I am going through lot of documentation for ACPI and PCIe. More I read, more I am getting confused. May be someone here will be able to guide me through about

a) How ACPI and PCIe are linked together? Are they completly separate entities?

b) Is ACPI only used for power management purposes? or it has additional purpose of hardware management?

c) Why ACPI if device registers can be used directly (using PCIe configuration address space) to manage power state (This may be dumb question).

d) If OS changes anything in PCIe configuration address space, it is OS job to keep ACPI tables in sync?

Note: My goal is to understand ACPI and how device driver can utilitise ACPI, for example, to get memory mapped information of a given device.

Regards

Tim_Roberts · March 28, 2012, 12:41pm

xxxxx@gmail.com wrote:

I am going through lot of documentation for ACPI and PCIe. More I read, more I am getting confused. May be someone here will be able to guide me through about

a) How ACPI and PCIe are linked together? Are they completly separate entities?

Yes. ACPI is merely a way for the BIOS to send structure information to
the operating system. ACPI is the way the BIOS tells the system which
devices are located where, what resources were assigned, how the bus
topologies are connected, what power capabilities they all have, and so
on. So, the PCIe topology will be reported using ACPI, but it’s just
one small part of the ACPI information.

ACPI plays its largest role at boot time, but it is also used for
communication with the BIOS during system operation. All of those Fn
keys on your laptop keyboard, for example, are transmitted via ACPI events.

b) Is ACPI only used for power management purposes? or it has additional purpose of hardware management?

It does play an additional role.

c) Why ACPI if device registers can be used directly (using PCIe configuration address space) to manage power state (This may be dumb question).

Because there are devices on your motherboard other than PCIe, some of
which do not have the configuration space concept.

d) If OS changes anything in PCIe configuration address space, it is OS job to keep ACPI tables in sync?

Note: My goal is to understand ACPI and how device driver can utilitise ACPI, for example, to get memory mapped information of a given device.

You would get that information from the operating system, not from
ACPI. The only time that a function driver needs to muck with ACPI is
when you have some custom event (“GPE”) hardwired into a motherboard
interface that is not passed through some existing bus.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Santhosh_Kumar-3 · March 28, 2012, 12:58pm

>>to get memory mapped information of a given device
Do:
EnumSystemFirmwareTables(‘ACPI’, pFirmwareTableBuffer, MAX_STRING_LENGTH); then do
strstr((char *)pFirmwareTableBuffer, “MCFG”) and make sure this succeeds. Then do
GetSystemFirmwareTable(‘ACPI’, ‘GFCM’, pFirmwareTableBuffer, MAX_STRING_LENGTH);
GFCM is MCFG which is the PCIE Config Base component.
Refer to the PCIE Firmware Spec (Not the PCIE 3.0 spec) and that has all the info you need to map the pFirmwareTableBuffer you get and there you will get the PCIE Config Base Address and that is the Memory Mapped address you can use.

Tim_Roberts · March 28, 2012, 1:10pm

xxxxx@gmail.com wrote:

>> to get memory mapped information of a given device
Do:
EnumSystemFirmwareTables(‘ACPI’, pFirmwareTableBuffer, MAX_STRING_LENGTH); then do
strstr((char *)pFirmwareTableBuffer, “MCFG”) and make sure this succeeds. Then do
GetSystemFirmwareTable(‘ACPI’, ‘GFCM’, pFirmwareTableBuffer, MAX_STRING_LENGTH);
GFCM is MCFG which is the PCIE Config Base component.
Refer to the PCIE Firmware Spec (Not the PCIE 3.0 spec) and that has all the info you need to map the pFirmwareTableBuffer you get and there you will get the PCIE Config Base Address and that is the Memory Mapped address you can use.

This is not good advice. All of that information is available to the
driver using the normal mechanisms. If the mapped address wasn’t handed
to you, then it’s not yours to use.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Santhosh_Kumar-3 · March 28, 2012, 1:36pm

OOps sorry Tim, I see his question was “a given device” and I gave him the wrong answer of finding the overall CfgBaseAdd. Mark I stand corrected please ignore it or just take it as an FYI.

Mark_Gupta · March 28, 2012, 11:08pm

> Yes. ACPI is merely a way for the BIOS to send structure information to

the operating system. ACPI is the way the BIOS tells the system which
devices are located where, what resources were assigned, how the bus
topologies are connected, what power capabilities they all have, and so
on. So, the PCIe topology will be reported using ACPI, but it’s just
one small part of the ACPI information.

o The other day I was trying to use PCIe Configuration Address Space to get device specific information (not using HalGetBusData). Would it be advisable to use PCIe Configuration Address Space directly (if one know’s how to use) or associated OS Halxxxx function? IO or MMIO information can be retrieved from BAR registers from device’s configuration space. Can it be used to instead?

o Does Halxxx function(s) use ACPI internally? As an example, windows kernel Hal function “HalTranslateBusAddress(…)” - would that be using ACPI or PCIe config address space to figure out translated address?

Jake_Oshins · March 29, 2012, 11:58am

I want a “like” button for Tim’s post.

Jake Oshins
Windows Kernel Team

This post implies no warrantees and confers no rights.

“Tim Roberts” wrote in message news:xxxxx@ntdev…

xxxxx@gmail.com wrote:

I am going through lot of documentation for ACPI and PCIe. More I read,
more I am getting confused. May be someone here will be able to guide me
through about

a) How ACPI and PCIe are linked together? Are they completly separate
entities?

Yes. ACPI is merely a way for the BIOS to send structure information to
the operating system. ACPI is the way the BIOS tells the system which
devices are located where, what resources were assigned, how the bus
topologies are connected, what power capabilities they all have, and so
on. So, the PCIe topology will be reported using ACPI, but it’s just
one small part of the ACPI information.

ACPI plays its largest role at boot time, but it is also used for
communication with the BIOS during system operation. All of those Fn
keys on your laptop keyboard, for example, are transmitted via ACPI events.

b) Is ACPI only used for power management purposes? or it has additional
purpose of hardware management?

It does play an additional role.

c) Why ACPI if device registers can be used directly (using PCIe
configuration address space) to manage power state (This may be dumb
question).

Because there are devices on your motherboard other than PCIe, some of
which do not have the configuration space concept.

d) If OS changes anything in PCIe configuration address space, it is OS
job to keep ACPI tables in sync?

Note: My goal is to understand ACPI and how device driver can utilitise
ACPI, for example, to get memory mapped information of a given device.

You would get that information from the operating system, not from
ACPI. The only time that a function driver needs to muck with ACPI is
when you have some custom event (“GPE”) hardwired into a motherboard
interface that is not passed through some existing bus.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Jake_Oshins · March 29, 2012, 12:00pm

It’s also worth noting that most chipsets claim that simultaneous access of
the MMCFG region by two processors is “undefined” meaning that it may, among
other things, result in a machine check. And you won’t be able to
synchronize with the PCI driver.

Please use the OS-supplied PCI configuration interfaces.

Jake Oshins
Windows Kernel Team

This post implies no warrantees and confers no rights.

“Tim Roberts” wrote in message news:xxxxx@ntdev…

xxxxx@gmail.com wrote:

>> to get memory mapped information of a given device
Do:
EnumSystemFirmwareTables(‘ACPI’, pFirmwareTableBuffer, MAX_STRING_LENGTH);
then do
strstr((char *)pFirmwareTableBuffer, “MCFG”) and make sure this succeeds.
Then do
GetSystemFirmwareTable(‘ACPI’, ‘GFCM’, pFirmwareTableBuffer,
MAX_STRING_LENGTH);
GFCM is MCFG which is the PCIE Config Base component.
Refer to the PCIE Firmware Spec (Not the PCIE 3.0 spec) and that has all
the info you need to map the pFirmwareTableBuffer you get and there you
will get the PCIE Config Base Address and that is the Memory Mapped
address you can use.

This is not good advice. All of that information is available to the
driver using the normal mechanisms. If the mapped address wasn’t handed
to you, then it’s not yours to use.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Tim_Roberts · March 29, 2012, 12:53pm

xxxxx@gmail.com wrote:

o The other day I was trying to use PCIe Configuration Address Space to get device specific information (not using HalGetBusData). Would it be advisable to use PCIe Configuration Address Space directly (if one know’s how to use) or associated OS Halxxxx function? IO or MMIO information can be retrieved from BAR registers from device’s configuration space. Can it be used to instead?

You should never access the configuration space directly. Configuration
space uses an indexed register approach, and because you don’t own that
resource, you and the operating system might get into a race condition
over the index.

A driver can access its own device’s configuration space using
IRP_MN_READ_CONFIG and IRP_MN_WRITE_CONFIG. There are samples on the
web. You shouldn’t be accessing the configuration space of devices you
don’t own.

o Does Halxxx function(s) use ACPI internally? As an example, windows kernel Hal function “HalTranslateBusAddress(…)” - would that be using ACPI or PCIe config address space to figure out translated address?

That depends on your definition. The PCI bus driver acquire all of this
information at boot time and caches it in memory and in the registry.
The HAL functions use the tables in memory to return this information.
The ACPI DSDT information does not change after boot, so there’s no need
to go diving back for that information. That’s a good thing, because
ACPI requests are slow.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

anton_bassov · March 29, 2012, 2:18pm

> You should never access the configuration space directly. Configuration space uses an

indexed register approach, and because you don’t own that resource, you and the operating system
might get into a race condition over the index.

Another point to consider is that some certain fields (for example, InterruptLine) directly obtained from PCI configuration space may be simply invalid - they get set by BIOS at the boot time but the OS subsequently reconfigures the machine…

Anton Bassov

Tim_Roberts · March 29, 2012, 2:47pm

xxxxx@hotmail.com wrote:

Another point to consider is that some certain fields (for example, InterruptLine) directly obtained from PCI configuration space may be simply invalid - they get set by BIOS at the boot time but the OS subsequently reconfigures the machine…

It can’t do that without also reprogramming the configuration space for
the affected devices.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

anton_bassov · March 29, 2012, 4:42pm

> It can’t do that without also reprogramming the configuration space for the affected devices.

Try to make an experiment, and you will see it with your own eyes (I did it under Linux, but the same may apply to Windows as well).

To make it even funnier for yourself, you can try to compare values of InterruptLine field from PCI configuration space against the values in MP configuration table (which, btw, is also considered a legacy mechanism that is not going to be used by Windows on ACPI-based system) - you will still see the difference. The thing is, the InterruptLine info that you get from PCI configuration space describes legacy IRQs and PIC, while the one in MP configuration table is all about IOAPIC pins…

Anton Bassov

Mark_Gupta · March 29, 2012, 6:44pm

> You should never access the configuration space directly. Configuration space uses an indexed register approach, and because you don’t own that resource, you and the operating system might get into a race condition over the index.

A driver can access its own device’s configuration space using IRP_MN_READ_CONFIG and IRP_MN_WRITE_CONFIG. There are samples on the web. You shouldn’t be accessing the configuration space of devices you don’t own.

Thanks a lot for clarifying. I now fully understand why to use IRP_MN_READ_CONFIG & IRP_MN_WRITE_CONFIG. I didn’t understand your point of “you and the operating system might get into a race condition over the index”. I am unable to get how race condition can happen if and only if driver read configuration space only.

Let me clarify what I mean

Assuming PCIe configuration space (MMIO) is at 0xE000_0000 (obtained from ACPI MCFG table).
If I am doing something like
DWORD dwRegisterVal = (DWORD)(*MMIOBaseAddress) // MMIOBaseAddress = 0xE000_0000.

How above code can lead to race condition? And above code will read VendorID and DeviceID for Bus:Device:Function of 0:0:0.

I am not proving you wrong here but I am trying to understand race condition part. Am I missing something here?

******************
Point 2
********************

> o Does Halxxx function(s) use ACPI internally? As an example, windows kernel Hal function “HalTranslateBusAddress(…)” - would that be using ACPI or PCIe config address space to figure out translated address?

That depends on your definition. The PCI bus driver acquire all of this information at boot time and caches it in memory and in the registry. The HAL functions use the tables in memory to return this information. The ACPI DSDT information does not change after boot, so there’s no need to go diving back for that information. That’s a good thing, because ACPI requests are slow.

Yesterday I was reading an article on how different buses are assigned bus addresses starting from Root Complex. And there it mentioned that Hot Plugging devices poses an issue because bus address range might change as resource requirements could not fit for a given bus and it has to be re-adjusted for that device to work. As a result, Bus Tree starting from Root Complex has to be re-configured/re-adjusted.

What you said is that “The PCI bus driver acquire all of this information at boot time and caches it in memory and in the registry”. Does that mean cache is invalidated when address window is re-adjusted for PCI-PCI/PCI-ISA bridges?

o One more question: Why information is stored in registry? Wouldn’t cache be enough as drivers will be using Halxxx functions which effectively will be use cache instead of Registry.

Mark_Gupta · March 29, 2012, 7:01pm

> Another point to consider is that some certain fields (for example, InterruptLine) directly obtained from PCI configuration space may be simply invalid - they get set by BIOS at the boot time but the OS subsequently reconfigures the machine…

From what I can get after reading documentation about PCI Header, InterruptLine register is not used by PCI devices at all. It is infact an informative register setup by BIOS for OS to use. And like you said, if OS reconfigures, there is no rule (I may be wrong here), to setup InterruptLine. Depends on OS whether to maintain InterruptLine in sync with what it has configured internally.

Well, IntPin is what matters. And I posted an thread about “$pir table” where I asked the question of IntPin and it’s mapping in IOAPIC. There could be Halxxx function for it but I want to know what vector or IRQL a pin is mapped to, how do I know? How PCIe interrupt pins are mapped to IOAPIC pins?

Tim_Roberts · March 29, 2012, 7:03pm

xxxxx@gmail.com wrote:

I didn’t understand your point of “you and the operating system might get into a race condition over the index”. I am unable to get how race condition can happen if and only if driver read configuration space only.

The standard mechanism for accessing configuration space is through the
CF8/CFC I/O ports. With those ports, you write the address you want in
CF8 and read or write the data in CFC. That has a race condition.

Let me clarify what I mean

Assuming PCIe configuration space (MMIO) is at 0xE000_0000 (obtained from ACPI MCFG table).

The ECAM (MMIO) mechanism is PCIExpress only. You’re correct that the
race does not happen here.

If I am doing something like
DWORD dwRegisterVal = (DWORD)(*MMIOBaseAddress) // MMIOBaseAddress = 0xE000_0000.

…after mapping E000_0000 to a virtual address. That region is a
resource you do not own. There’s little risk, but it’s still not kosher.

How above code can lead to race condition? And above code will read VendorID and DeviceID for Bus:Device:Function of 0:0:0.

…after you map E000_0000 to a virtual address.

What you said is that “The PCI bus driver acquire all of this information at boot time and caches it in memory and in the registry”. Does that mean cache is invalidated when address window is re-adjusted for PCI-PCI/PCI-ISA bridges?

Yes, this is called “rebalancing”. PCI hotplugging was not possible
until Vista, but after that point, the PCI bus driver does have the
ability to “rebalance” by re-assigning resources. I don’t think the
BIOS or ACPI gets involved in that process, but I’ve never had to worry
about that.

o One more question: Why information is stored in registry? Wouldn’t cache be enough as drivers will be using Halxxx functions which effectively will be use cache instead of Registry.

The registry allows that information to be read by other clients in a
safe and supported way. The HKLM\HARDWARE registry hive is all stored
in memory anyway – it’s somewhat analogous to the procfs or sysfs
concept in Linux.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Tim_Roberts · March 29, 2012, 7:14pm

xxxxx@gmail.com wrote:

Well, IntPin is what matters. And I posted an thread about “$pir table” where I asked the question of IntPin and it’s mapping in IOAPIC. There could be Halxxx function for it but I want to know what vector or IRQL a pin is mapped to, how do I know? How PCIe interrupt pins are mapped to IOAPIC pins?

You don’t need to know this. Your motherboard might not even HAVE an
IOAPIC. Let the kernel worry about the implementation. You just follow
the abstraction and call IoConnectInterrupt. The kernel and the HAL
will program the hardware to make that connection happen.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Mark_Gupta · March 29, 2012, 7:20pm

“the PCI bus driver does have the ability to “rebalance” by re-assigning resources”

And what about device driver? If device driver has obtained resource information at the time of start-up and assuming re-balance happens after that, does device driver gets notified to re-configure itself because of re-balancing act?

Thinking about it further, shouldn’t re-balancing act (don’t know how it works) disables device interrupts as device drivers may need to be re-configured? I might be going out of topic here but this seems to be interesting topic to discuss.

Doron_Holan · March 29, 2012, 7:24pm

The driver will get a query stop and then stop pnp irp sequence. During stop processing you disable your interrupt and stop touching hw, basically just like your power down oath. Then you will get another start irp where you get you new resources, connect and enable interrupts, etc. In kmdf this is abstracted around power down/up and evtdevicerelease/preparehardware

d

debt from my phone

From: xxxxx@gmail.com
Sent: 3/29/2012 4:20 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] How ACPI and PCIe are linked together

“the PCI bus driver does have the ability to “rebalance” by re-assigning resources”

And what about device driver? If device driver has obtained resource information at the time of start-up and assuming re-balance happens after that, does device driver gets notified to re-configure itself because of re-balancing act?

Thinking about it further, shouldn’t re-balancing act (don’t know how it works) disables device interrupts as device drivers may need to be re-configured? I might be going out of topic here but this seems to be interesting topic to discuss.

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Mark_Gupta · March 29, 2012, 7:50pm

Heaps thanks Doron and Tim for improving my knowledge

Maxim_S_Shatskih · March 30, 2012, 2:59am

>up and assuming re-balance happens after that, does device driver gets notified to re-configure itself

because of re-balancing act?

Surely yes.

Thinking about it further, shouldn’t re-balancing act (don’t know how it works) disables device
interrupts as device drivers may need to be re-configured?

Yes, STOP is sent to the driver, and the driver must disable the interrupts.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com