What you’re doing, Sagi Bar, isn’t legal within the PCIe specs. You could be causing bus errors and other things with odd side effects. And since most commodity client machines won’t trigger a machine check when these happen, results at that point are really undefined. In a machine costing somewhat more, you’ll tend to see machine checks and blue screens when you do this.
The only supported way to do hot-plug on PCIe involves a root port (or other downstream port, though these are rare) which implements the standard PCIe hotplug controller. Then there’s a sequence of states that brings down the link in a controlled fashion and then brings it up again later.
As for explaining the behavior you saw, I think that the only thing that I’d add to Tim’s explanation is that in the second case, where you ripped the device out while it was live and then put it back again, you probably triggered a surprise-removal sequence followed by a re-add of the device, and the second driver stack came up before the first one released its resources. Because you did this in an unsupported fashion, you hung the first stack and it never unwound.
– Jake Oshins
Windows Kernel Team
(former PnP guy)
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Tim Roberts
Sent: Monday, March 10, 2014 10:10 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Windows and PCI resources
xxxxx@walla.co.il wrote:
One interesting thing I can’t explain is how are MMIO BAR addresses are assigned by Windows.
Example:
Power up both the card and the PC. The card is an AHCI controller so it has MMIO at bar #5. AHCI controller is recognized and loaded by Windows. BAR5 is assigned address 0xF7C20000.
Experiment#1
1.Disable the AHCI driver.
2. Switch off - on the card
Result
Same BAR#5 value is restored (0xF7C20000)
!Question! : where does Windows keep the previous value and how does it know that the value should be reassigned to this particular device
When your machine boots, the BIOS allocates a small hole in physical
memory just below the 4GB boundary to contain all of your memory-mapped
PCI devices. The size of that hole depends on the BIOS, but it ranges
from 256MB to 2 GB. Let’s assume you have 8GB of RAM, and your BIOS
uses a 512MB hole. So, you’ll have RAM at physical addresses from
00000000 to E0000000, then an unassigned hole, then RAM from 100000000 to 21FFFFFFF.
The BIOS then does an initial run, assigning physical addresses to your
devices with BARs. All of the devices go into that hole.
The use of that space does not change very often. Usually, it does not
change at all. Because there is no churn, when an address frees up, the
next device that comes along is going to get assigned to the same location.
!Question!
Why is the device assigned a new BAR#5 address?
Why is the behavior inconsistent?
Why is device manager stuck.
I ask not of pure curiousity, but out of desire to understand how Windows actually handles PCI resource reassignment upon hot swap.
I have tried looking for Microsoft documents in this regard, but found nothing thorough.
The answer, really, is “because”. This is an implementation detail that
does not matter to anyone.
What operating system were you testing? Hot-plug PCI support has not
been around all that long. It requires special support in the BIOS,
special notification mechanisms in ACPI, and at least Vista. Remember
that the original PCI specifications did not support hot-plugging at
all. Once a device was gone, it was gone forever.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
NTDEV is sponsored by OSR
Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer