One knotty problem on 64-bit Memory Bar in PCI Device booting to Windows Xp

Hi All,

I met a problem when I tried to enable a PCI device in Windows XP Sp2
32bit version. This device has a 64bit MMIO Bar. Taken into
consideration that I have only limited memory plugged in, I only
allocate a 32bit address to this device and set the upper 32bit address
to 0h in bios. But after I boot to the XP, Xp will write some address in
the upper 32bit address. Then when I later viewed the device in device
manager, Xp will report error because of out of resource.

Some data collected are: before Xp boot, the device MMIO bar is
00000000902A6904h. after Xp boot, the device MMIO bar 48000010902A6904h.
The upper 32bit are not supposed to be changed. But Xp wrote it anyway.

I tried it in Windows 2000 and there is no such issue. Windows 2003
behavior is just the same as Windows Xp

So what does Windows Xp do when it comes to resource allocation
especially facing 64bit Bar? Any insight is greatly appreciated. Thanks!

Best regards,

Cody

Wu, Cody wrote:

I met a problem when I tried to enable a PCI device in Windows XP Sp2
32bit version. This device has a 64bit MMIO Bar. Taken into
consideration that I have only limited memory plugged in, I only
allocate a 32bit address to this device and set the upper 32bit
address to 0h in bios.

Why do you think the BAR has anything to do with “limited memory”? BAR
values are physical addresses. As long as you have a 64-bit PCI bus,
the exact value shouldn’t make a difference.

But after I boot to the XP, Xp will write some address in the upper
32bit address. Then when I later viewed the device in device manager,
Xp will report error because of out of resource.

Some data collected are: before Xp boot, the device MMIO bar is
00000000902A6904h. after Xp boot, the device MMIO bar
48000010902A6904h. The upper 32bit are not supposed to be changed. But
Xp wrote it anyway.

There is nothing that says XP can’t rewrite the BAR to whatever it wants.

However, that is an unusual value for a BAR address. How are you
determining the BAR value? Usually, a BAR will be assigned closer to
the top of physical memory. Also, this value is only aligned to an
8-byte boundary. How large is the memory region supposed to be? A
256-byte BAR will always be aligned on a 256-byte address, because the
lower 8-bits of the BAR register are not writable.

Does the device also have a 32-bit BAR?


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Tim -

Looks to me like the BAR is aligned on a 256 byte boundary. The trailing
4 is the read-only bit that says it’s a 64-bit address.

When I first read his post, for some reason it escaped me that he might be
working on the System BIOS code that is assigning resources. I now assume
that he’s working on the System BIOS code.

I would guess that Windows is rewriting the BARs because it doesn’t like
something it sees in the configuration of the bridges leading to this
card. I saw a case where the BIOS left the MemoryBase register on a
bridge at 0 and Windows reconfigured that bridge and everything on the
other side of it. (How this would ever work with the MemoryBase at 0 is
beyond me, except that this must have been some strange kind of bridge -
every access on the secondary side of the bridge to low addresses should
not have been forwarded to the primary bus.) Anyway, once the System BIOS
was fixed and the proper value was placed in the MemoryBase register of
the bridge, Windows stopped reconfiguring and everything worked.

Jerry.

Tim Roberts
Sent by: xxxxx@lists.osr.com
02/01/2007 03:10 PM
Please respond to
“Windows System Software Devs Interest List”

To
“Windows System Software Devs Interest List”
cc

Subject
Re: [ntdev] One knotty problem on 64-bit Memory Bar in PCI Device booting
to Windows Xp

Wu, Cody wrote:
>
>
>
> I met a problem when I tried to enable a PCI device in Windows XP Sp2
> 32bit version. This device has a 64bit MMIO Bar. Taken into
> consideration that I have only limited memory plugged in, I only
> allocate a 32bit address to this device and set the upper 32bit
> address to 0h in bios.
>

Why do you think the BAR has anything to do with “limited memory”? BAR
values are physical addresses. As long as you have a 64-bit PCI bus,
the exact value shouldn’t make a difference.

> But after I boot to the XP, Xp will write some address in the upper
> 32bit address. Then when I later viewed the device in device manager,
> Xp will report error because of out of resource.
>
>
>
> Some data collected are: before Xp boot, the device MMIO bar is
> 00000000902A6904h. after Xp boot, the device MMIO bar
> 48000010902A6904h. The upper 32bit are not supposed to be changed. But
> Xp wrote it anyway.
>

There is nothing that says XP can’t rewrite the BAR to whatever it wants.

However, that is an unusual value for a BAR address. How are you
determining the BAR value? Usually, a BAR will be assigned closer to
the top of physical memory. Also, this value is only aligned to an
8-byte boundary. How large is the memory region supposed to be? A
256-byte BAR will always be aligned on a 256-byte address, because the
lower 8-bits of the BAR register are not writable.

Does the device also have a 32-bit BAR?


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Hi Tim,

Thanks for your quick response. I’d like to provide my opinions on the
points you have put up.:slight_smile:

Why do you think the BAR has anything to do with “limited memory”? BAR
values are physical addresses. As long as you have a 64-bit PCI bus,
the exact value shouldn’t make a difference.

I should say 64bit BAR actually has nothing to do with 64-bit PCI Bus.
You can have 64bit Bar in either 32bit or 64bit bus. When I say “Limited
Memory”, I mean the physical memory space is shrank when you plug-in
less memory. As you have pointed out, BARs are programmed with physical
address; it wouldn’t make sense for XP to program it with any address
higher than 4G, i.e. non-zero value in higher 32 bit.

There is nothing that says XP can’t rewrite the BAR to whatever it
wants.

Well, XP actually is limited in its capability to do so even if it can.
XP is designed such that it should respect Bios’s resource allocation
unless something goes wrong. Later operating systems are indeed endowed
with this capability which is termed “Multi-level Rebalance” in
Microsoft terms. As I haven’t noticed anything wrong in resource
allocation scheme, it puzzles me most that XP has given me a
non-existent physical address.

How are you determining the BAR value?
We determine the value in BIOS in much the similar way as you pointed
out. As we are developing BIOS code, we just follow the normal flow to
assign resources. The trouble-making device requires merely 16 byte.

Does the device also have a 32-bit BAR?

Unfortunately not. Only 1 IOBAR. And in Win2K the device manager shows 2
Memory resources have been allocated which make me doubt whether Win2K
cannot recognize the 64bit BAR.

Best regards,
Cody
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tim Roberts
Sent: Friday, February 02, 2007 4:08 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] One knotty problem on 64-bit Memory Bar in PCI
Device booting to Windows Xp

Wu, Cody wrote:

I met a problem when I tried to enable a PCI device in Windows XP Sp2
32bit version. This device has a 64bit MMIO Bar. Taken into
consideration that I have only limited memory plugged in, I only
allocate a 32bit address to this device and set the upper 32bit
address to 0h in bios.

Why do you think the BAR has anything to do with “limited memory”? BAR
values are physical addresses. As long as you have a 64-bit PCI bus,
the exact value shouldn’t make a difference.

But after I boot to the XP, Xp will write some address in the upper
32bit address. Then when I later viewed the device in device manager,
Xp will report error because of out of resource.

Some data collected are: before Xp boot, the device MMIO bar is
00000000902A6904h. after Xp boot, the device MMIO bar
48000010902A6904h. The upper 32bit are not supposed to be changed. But
Xp wrote it anyway.

There is nothing that says XP can’t rewrite the BAR to whatever it
wants.

However, that is an unusual value for a BAR address. How are you
determining the BAR value? Usually, a BAR will be assigned closer to
the top of physical memory. Also, this value is only aligned to an
8-byte boundary. How large is the memory region supposed to be? A
256-byte BAR will always be aligned on a 256-byte address, because the
lower 8-bits of the BAR register are not writable.

Does the device also have a 32-bit BAR?


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Hi Jerry,

Thanks for your reply.

Yes, I am working on the System Bios code for some newly debuted
platform. :slight_smile:

I also think XP have the alternative to do so if something really goes
wrong. But our device is at Bus 0 so no Bridge is ever involved and
that adds to my puzzle further. Little information that helps can be
gleaned from Microsoft website. Normally everything works just fine but
things get difficult when problems do present themselves.

Best regards,

Cody


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@attotech.com
Sent: Friday, February 02, 2007 4:47 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] One knotty problem on 64-bit Memory Bar in PCI
Device booting to Windows Xp

Tim -

Looks to me like the BAR is aligned on a 256 byte boundary. The
trailing 4 is the read-only bit that says it’s a 64-bit address.

When I first read his post, for some reason it escaped me that he might
be working on the System BIOS code that is assigning resources. I now
assume that he’s working on the System BIOS code.

I would guess that Windows is rewriting the BARs because it doesn’t like
something it sees in the configuration of the bridges leading to this
card. I saw a case where the BIOS left the MemoryBase register on a
bridge at 0 and Windows reconfigured that bridge and everything on the
other side of it. (How this would ever work with the MemoryBase at 0 is
beyond me, except that this must have been some strange kind of bridge -
every access on the secondary side of the bridge to low addresses should
not have been forwarded to the primary bus.) Anyway, once the System
BIOS was fixed and the proper value was placed in the MemoryBase
register of the bridge, Windows stopped reconfiguring and everything
worked.

Jerry.

Tim Roberts
Sent by: xxxxx@lists.osr.com

02/01/2007 03:10 PM

Please respond to
“Windows System Software Devs Interest List”

To

“Windows System Software Devs Interest List”

cc

Subject

Re: [ntdev] One knotty problem on 64-bit Memory Bar in PCI Device
booting to Windows Xp

Wu, Cody wrote:
>
>
>
> I met a problem when I tried to enable a PCI device in Windows XP Sp2
> 32bit version. This device has a 64bit MMIO Bar. Taken into
> consideration that I have only limited memory plugged in, I only
> allocate a 32bit address to this device and set the upper 32bit
> address to 0h in bios.
>

Why do you think the BAR has anything to do with “limited memory”? BAR
values are physical addresses. As long as you have a 64-bit PCI bus,
the exact value shouldn’t make a difference.

> But after I boot to the XP, Xp will write some address in the upper
> 32bit address. Then when I later viewed the device in device manager,
> Xp will report error because of out of resource.
>
>
>
> Some data collected are: before Xp boot, the device MMIO bar is
> 00000000902A6904h. after Xp boot, the device MMIO bar
> 48000010902A6904h. The upper 32bit are not supposed to be changed. But
> Xp wrote it anyway.
>

There is nothing that says XP can’t rewrite the BAR to whatever it
wants.

However, that is an unusual value for a BAR address. How are you
determining the BAR value? Usually, a BAR will be assigned closer to
the top of physical memory. Also, this value is only aligned to an
8-byte boundary. How large is the memory region supposed to be? A
256-byte BAR will always be aligned on a 256-byte address, because the
lower 8-bits of the BAR register are not writable.

Does the device also have a 32-bit BAR?


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

— Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256 To unsubscribe, visit the
List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer