MmMapIoSpace providing wrong address during boot?

NTDEV
1

Hello Gents,

We are encountering a problem with our recent hardware using PCIe instead
PCI. The driver is actually mapping BAR0 using MmMapIoSpace().

Actually everything goes well if we install the driver after boot. The
problem occurs on some machine (15% of the machines) only when the driver
“loads” during boot. I’ve been step tracing this piece of code and
MmMapIoSpace() is returning an address for a given physical address. The
physical address is constant in my case, I verified that loading the driver
after boot or auto start during boot. The address returned by MmMapIoSpace()
before boot does not point to the expected physical memory. This address
actually points to a 0xFFFFFFFF area. Note that the device is displayed
correctly in the device manager.

Then I can proceed with this sequence :

  1. Uninstall the device in the device manager

  2. Reset the bridge ( the bridge our card connects to ) using PCI tree.

  3. Scan for hardware changed in the device manager

  4. Install the exact same driver

At this time the BAR reads perfectly. Well of course address returned by
MmMapIoSpace() is different each time but this is expected I guess. We have
never seen such a problem so far before our PCIe card. And this problem is
happening only on some machines, once again.

I’ve tried to get our driver to load last using the SubOrderGroup but the
same problem occurs. On every boot, the address returned by MmMapIoSpace()
seems wrong.

The way that function is called :

DevExt->RegsBase = (PUCHAR) MmMapIoSpace( regsBasePA,

regsLength,

MmNonCached );

Did you guys have seen that already, any pointers?

With my best regards,

Arnaud

2

Yikes.

Is this specific to any particular OS, architecture, version?

Peter
OSR

3

> The problem occurs on some machine (15% of the machines) only when

the driver “loads” during boot.

Smells of a BIOS and/or hardware/timing problem.
Could this be the case?

4

When are you doing the MmMapIoSpace call?

Larry C

5

Sorry I omitted the most fundamental info. We have been doing test on 32
bits architecture XP SP3 as well as 7. The faulty vectors have different
PCI bridge.

We have never seen this problem before on our PCI hardware, this is
something related to PCIe I assume. Actually same machine, same driver,
same OS. Difference is the interface layer between the processing FPGA;a
PCIe core instead of a PCI core also running on FPGA. In the other hand
same PCIe hardware, same driver and same OS on another machine works well.
I’ve confirmed XP/7 using double boot on the faulty vectors nothing
indicates it is os dependant.

Big thanks,

Arnaud

Yikes.

Is this specific to any particular OS, architecture, version?

Peter
OSR

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

6

We converted our PCI card to PCIe last year without any problems. I converted to WDF and the driver runs fine on XP, Vista, Win7 and Win7-64bit. I do the MmMapIoSpace in EvtDeviceD0Entry(). Maybe your card takes longer to get to D0 power state on some machines.

Larry C

7

Hello Larry,

I am calling MmMapIoSpace in EvtDevicePrepareHardware as recommended by MSDN
:

“.EvtDevicePrepareHardware, which delivers the device’s system-assigned
resources to the driver. The driver can perform operations, such as mapping
the device’s bus-relative memory into the processor’s virtual address space,
that make the hardware accessible to the driver.”

I’ve been looking a little about it and of course my event is called before
“EvtDeviceD0Entry”. I am going to move my code there and let you know if it
makes any difference.

Thank you,

Arnaud

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@honeywell.com
Sent: Tuesday, June 22, 2010 9:30 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] MmMapIoSpace providing wrong address during boot?

We converted our PCI card to PCIe last year without any problems. I
converted to WDF and the driver runs fine on XP, Vista, Win7 and Win7-64bit.
I do the MmMapIoSpace in EvtDeviceD0Entry(). Maybe your card takes longer to
get to D0 power state on some machines.

Larry C

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

8

You have the call in the right place, keep it in preparehw().

d

dent from a phpne with no keynoard

-----Original Message-----
From: Arnaud Maye
Sent: June 23, 2010 12:21 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] MmMapIoSpace providing wrong address during boot?

Hello Larry,

I am calling MmMapIoSpace in EvtDevicePrepareHardware as recommended by MSDN
:

“.EvtDevicePrepareHardware, which delivers the device’s system-assigned
resources to the driver. The driver can perform operations, such as mapping
the device’s bus-relative memory into the processor’s virtual address space,
that make the hardware accessible to the driver.”

I’ve been looking a little about it and of course my event is called before
“EvtDeviceD0Entry”. I am going to move my code there and let you know if it
makes any difference.

Thank you,

Arnaud

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@honeywell.com
Sent: Tuesday, June 22, 2010 9:30 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] MmMapIoSpace providing wrong address during boot?

We converted our PCI card to PCIe last year without any problems. I
converted to WDF and the driver runs fine on XP, Vista, Win7 and Win7-64bit.
I do the MmMapIoSpace in EvtDeviceD0Entry(). Maybe your card takes longer to
get to D0 power state on some machines.

Larry C

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

9

Just a dumb question …

Is the Memory Space Enable bit set in the PCI Command Register at the time
you observe this not working correctly?

“Arnaud Maye”
Sent by: xxxxx@lists.osr.com
06/22/2010 09:34 AM
Please respond to
“Windows System Software Devs Interest List”

To
“Windows System Software Devs Interest List”
cc

Subject
[ntdev] MmMapIoSpace providing wrong address during boot?

Hello Gents,

We are encountering a problem with our recent hardware using PCIe instead
PCI. The driver is actually mapping BAR0 using MmMapIoSpace().

Actually everything goes well if we install the driver after boot. The
problem occurs on some machine (15% of the machines) only when the driver
?loads? during boot. I?ve been step tracing this piece of code and
MmMapIoSpace() is returning an address for a given physical address. The
physical address is constant in my case, I verified that loading the
driver after boot or auto start during boot. The address returned by
MmMapIoSpace() before boot does not point to the expected physical memory.
This address actually points to a 0xFFFFFFFF area. Note that the device is
displayed correctly in the device manager.

Then I can proceed with this sequence :

1) Uninstall the device in the device manager
2) Reset the bridge ( the bridge our card connects to ) using PCI
tree.
3) Scan for hardware changed in the device manager
4) Install the exact same driver

At this time the BAR reads perfectly. Well of course address returned by
MmMapIoSpace() is different each time but this is expected I guess. We
have never seen such a problem so far before our PCIe card. And this
problem is happening only on some machines, once again.

I?ve tried to get our driver to load last using the SubOrderGroup but the
same problem occurs. On every boot, the address returned by MmMapIoSpace()
seems wrong.

The way that function is called :

DevExt->RegsBase = (PUCHAR) MmMapIoSpace( regsBasePA,
regsLength,
MmNonCached );

Did you guys have seen that already, any pointers?

With my best regards,

Arnaud

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

10

We had problems with certain DELL boxes like the 5400, where the driver would occasionally Blue Screen at boot time. One of Microsoft’s driver gurus had me move the final initialization of our PCIe card to EvtDeviceD0Entry(), this included the mapping of the card’s memory. Everything worked fine after that.

Larry C

11

On boot there is no difference between these 2 callbacks, they are called one after the other. Making this change should not solve the issue. Why did this fix the issue?

The problem with making the call in d0entry() is that you must unmap in d0exit() which is called on every Sx transition, while if you do it in preparehw(), you unmap in releasehw() which is only called rebalance and device removal.

d

dent from a phpne with no keynoard

-----Original Message-----
From: xxxxx@honeywell.com
Sent: June 23, 2010 8:38 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] MmMapIoSpace providing wrong address during boot?

We had problems with certain DELL boxes like the 5400, where the driver would occasionally Blue Screen at boot time. One of Microsoft’s driver gurus had me move the final initialization of our PCIe card to EvtDeviceD0Entry(), this included the mapping of the card’s memory. Everything worked fine after that.

Larry C

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

12

On 23/06/2010 17:22, Doron Holan wrote:

On boot there is no difference between these 2 callbacks, they are called one after the other. Making this change should not solve the issue. Why did this fix the issue?

To elaborate further (Doron is a busy man):

IIRC -

Prepare / Release HW - Read resources, and in PCI drivers, PCI config.
Allocate / Map resources.
DO Entry / Exit: Access device through mapped bar’s. Set up and tear
down device. Save any info that the device can’t retain over Dx power
state transitions.

EvtDeviceAdd - One off allocation of per device WDF objects not related
to hardware that you can do before knowing much about the hardware
config (e.g. Device interfaces, WMI providers etc etc). - normally
parenting rules do most of the cleanup for you.

Gotchas:

WDF objects that get allocated in EvtPrepareHardware. Remember to free
them in EvtReleaseHardware. If you forget to do this, it will work
*almost all* of the time - Almost all calls to EvtReleaseHardware are
followed by device destruction. The one gotcha is PnP rebalance, which
happens rarely. You can have a driver work fine for ages, and then it
fails the PnPDTest or WHQL, normally with an interesting shade of blue.

MH.

13

On 23/06/2010 17:22, Doron Holan wrote:

On boot there is no difference between these 2 callbacks, they are called one after the other. Making this change should not solve the issue. Why did this fix the issue?

Oh yes - two more thoughts:

  • Things are a bit different for drivers marked as boot start - but
    normally only in terms of driver entry happening earlier - device
    enumeration is normally OK.
  • Is he perhaps passing parameters into calls from a registry read that
    might well fail at earlier phases of the boot?

MH.

14

Are you mapping raw or translated resources?

15

Boot start vs dynamic start will no effect on the timing between calling PrepareHW() and D0Entry, boot start will only effect the timing between DriverEntry() and AddDevice()

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Martin Harvey
Sent: Wednesday, June 23, 2010 9:36 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] MmMapIoSpace providing wrong address during boot?

On 23/06/2010 17:22, Doron Holan wrote:

On boot there is no difference between these 2 callbacks, they are called one after the other. Making this change should not solve the issue. Why did this fix the issue?

Oh yes - two more thoughts:

  • Things are a bit different for drivers marked as boot start - but normally only in terms of driver entry happening earlier - device enumeration is normally OK.
  • Is he perhaps passing parameters into calls from a registry read that might well fail at earlier phases of the boot?

MH.

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

16

Hello Guys,

Well actually I’ve figured out that after boot, the PCIe bridge ( between
the ICH10 and our device ) is not properly configured. Most of the PCI
configuration space is zeroed. A bridge should have a few entries there, one
per link on the secondary bus I believe. After the bridge is reset using PCI
tree the bridge is then properly configured and we can communicate with our
card.

Am not sure what is different between Windows enumerating the BUS and
resetting bridge using PCI tree ( as seen from the bridge ).

The more it goes the less this seems to be a software problem. I guess
something goes wrong between our device and the bridge during first
enumeration and this makes the bridge to believe no devices is on the
secondary bus. This of course does not explain why Windows detect the device
having the bridge left non configured. Actually this 0xFFFFFFFF is likely to
be a communication time out.

I guess I am going to send the case back to our firmware department.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Doron Holan
Sent: Wednesday, June 23, 2010 8:46 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] MmMapIoSpace providing wrong address during boot?

Boot start vs dynamic start will no effect on the timing between calling
PrepareHW() and D0Entry, boot start will only effect the timing between
DriverEntry() and AddDevice()

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Martin Harvey
Sent: Wednesday, June 23, 2010 9:36 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] MmMapIoSpace providing wrong address during boot?

On 23/06/2010 17:22, Doron Holan wrote:

On boot there is no difference between these 2 callbacks, they are called
one after the other. Making this change should not solve the issue. Why did
this fix the issue?

Oh yes - two more thoughts:

  • Things are a bit different for drivers marked as boot start - but normally
    only in terms of driver entry happening earlier - device enumeration is
    normally OK.
  • Is he perhaps passing parameters into calls from a registry read that
    might well fail at earlier phases of the boot?

MH.

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

17

On 24/06/2010 08:57, Arnaud Maye wrote:

Am not sure what is different between Windows enumerating the BUS and
resetting bridge using PCI tree ( as seen from the bridge ).

I don’t know the details, but in some cases windows leaves the
initialization done by the BIOS, and in some cases it does it itself.

Do you get different results if you disable/enable that chunk of the PCI
bus heirarchy in device manager? If you hibernate / resume the machine?

This of course does not explain why Windows detect the device
having the bridge left non configured.

Because there are rules about whether windows attempts to set up PCI
hardware itself, or whether it leaves any pre-existing BIOS configuration.

MH.