Requesting 256GB PCIe BAR

Hi
We developing PCIe based device that will be used in Win7 x64 and Win10 x64 systems.
One of the BARs of this device need to be 256GB length.

The Max size that i actually managed to define, that windows successfully allocated resources for it is - 1GB. When asking BAR length bigger tan that, i see in device PCIe configuration space that BAR was not assigned address in virtual space, and the driver had error :“… insufficient resources …”

Is there limitation on BAR size in Windows ?

thanks ahead
I will appreciate any help with that issue

This is very ill-conceived design. Why do you need to map your whole volume to PCI space?

Lets assume for the moment that there is no other way, and we “MUST” to do so in order to be able to solve some requirements. (i just want to skip the long explanation about tracking algorithm that implemented on that device, and why it should expose this size of memory and not less)

So assume that i need to request Very large Address space (256GB) on BAR0 of the device we developing, is Windows x64 system (that it’s System virtual space is 8TB) will block it ? Is some limitations exists in Windows pcie bus driver?

thanks

If requirements force you to use bad design which is quite likely to have compatibility problems, then you need seriously think about renegotiating those requirements.

It’s very possible that PCI enumerators in Window and Linux do not accommodate insane designs and will not enumerate bars over 4GB (or even over 2GB).

32 bit OS builds still exist, and your device will not work for them. Now what?

xxxxx@sandisk.com wrote:

Lets assume for the moment that there is no other way, and we “MUST” to do so in order to be able to solve some requirements. (i just want to skip the long explanation about tracking algorithm that implemented on that device, and why it should expose this size of memory and not less)

So assume that i need to request Very large Address space (256GB) on BAR0 of the device we developing, is Windows x64 system (that it’s System virtual space is 8TB) will block it ? Is some limitations exists in Windows pcie bus driver?

It’s entirely possible that this is a limitation in your BIOS or your
PCIe root complex. The BIOS does BAR assignments at boot time. Do you
have a relatively high-end motherboard, or is it a cheap one? Do you
know which chipset it uses? I don’t know how many BIOSes can assign
physical address ranges outside of the “PCIe hole” below 4GB. Do you
see your device getting assigned to a physical address at boot time?


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

> PCIe root complex. The BIOS does BAR assignments at boot time.

All BARs? or for boot disk controller only?


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

Maxim S. Shatskih wrote:

> PCIe root complex. The BIOS does BAR assignments at boot time.
All BARs? or for boot disk controller only?

The BIOS assigns ALL BARs at boot time. It can’t be sure you’ll be
running an operating system that is competent enough to handle that
task. Windows didn’t start overriding the BIOS assignments until the
21st Century.

Now, it might be that some (most?) BIOSes skip the huge BARs. I’ve
never encountered one to know for sure.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Once I worked with machines that have built-in VME bridges and custom BIOS that allocates 4 or 8 GB for the bridge. A “normal” PCI driver for the bridge loads there without any additional tricks.

(Unfortunately my previous reply disappeared… Could it be because it referred to Linuxbios project?)

– pa

hi, thanks for replying
I’ll r to answer couple of questions in the same post

  1. This system intended only for internal use by our company and will run only on Win7 x64 and Win10 x64 (no 32bit support is ever required)

  2. When asking 2GB of memory for BAR0 , i see that the BIOS or/and Windows do not assign physical memory for that BAR at all ( it’s value is 0x0).

  3. the Huge (256GB) address required for 1 BAR only

  4. I read that there is “re-mapping” enable/disable flag in BIOS that allow to OS to remap BARs assigned by BIOS. I did’t see this option in our BIOS (but it still may be enabled by default, i checking it via BIOS manufacture support)

I will appreciate any assistance

Uri

>1. This system intended only for internal use by our company and will run

only on Win7 x64 and Win10 x64 (no 32bit support is ever required)

256GB is even beyond the maximum physical memory that Win7 x64 can handle,
let alone the maximum BAR size.

//Daniel

I see that you simply want an ease of access to your memory module under test.
You can achieve the same by having a control register which will remap offset in a smaller BAR to large offset in your memory module.

The kernel virtual space limit is 8 TB or more, so the OP should be able to get 256 GB of I/O space mapped?

Here: http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778(v=vs.85).aspx

– pa

xxxxx@fastmail.fm wrote:

The kernel virtual space limit is 8 TB or more, so the OP should be able to get 256 GB of I/O space mapped?

Only if the I/O space is already mapped into physical address space. At
this point, THAT’S not happening.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Hi
Thanks for pointing me to physical memory limitations (here - http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778(v=vs.85).aspx )

I want to clarify - what is EXACT definition of “physical memory” in PC?

  1. Is it RAM only ?
  2. or RAM + “memory” exposed by different devices ?
  3. or something else…

According to above link - Win7 x64 Enterprise Physical memory is limited to 192 GB , thus it seems that i can’t define more then that in my Device.
But In Win10 x64 Enterprise , the limit is 2TB –> so it should be OK from OS point of view ?

(256GB should be Memory mapped and not I/O)

Uri.

> I want to clarify - what is EXACT definition of “physical memory” in PC?

  1. Is it RAM only ?

In the sense of these limits - RAM only.
PCI BARs are mapped as I/O space and so are outside of the RAM.
Page tables are still needed to map the I/O space, and 256 GB requires a lot of PTEs - unless Windows has some optimization for this, such as large pages.
But what does limit you is the PCI driver, it manages allocation of I/O space for BARs, and configuration of all PCI bridges on the way to the device.

– pa

> I want to clarify - what is EXACT definition of “physical memory” in PC?

I would define it as “all the regions that are marked as ‘System memory’ in a map that firmware presents to the OS upon the boot”. Memory-mapped BARs are not included in it, so that I would not include them in this definition…

Anton Bassov

Uri, were you able to resolve this?

@Uri said:
Hi
We developing PCIe based device that will be used in Win7 x64 and Win10 x64 systems.
One of the BARs of this device need to be 256GB length.

The Max size that i actually managed to define, that windows successfully allocated resources for it is - 1GB. When asking BAR length bigger tan that, i see in device PCIe configuration space that BAR was not assigned address in virtual space, and the driver had error :“… insufficient resources …”

Is there limitation on BAR size in Windows ?

thanks ahead
I will appreciate any help with that issue

Uri, May i know if you were able to resolve this? If so, how?

Don’t necropost. Just don’t.

Cool topic… start a new thread if you wish.