low-level PCI info

Hi all,

I’ve written PCI drivers for various pieces of hardware and never really cared how the magic happened to convert my accesses to/from the memory-mapped address spaces into signals on the PCI bus. I did a “data = *(address)” and was a happy developer. Of course, you all see where this is going … I now need to dig into the magic and turn it into knowledge. Any pointers on where I can find this info? Papers, articles, example source code, what pieces are in seperate drivers versus inside the kernel and hal versus inside the BIOS, … I’m not sure where to start looking. I’m familiar with how the signals should look on the actual PCI bus - I’m looking for the conversion from a memory-mapped access (or series of accesses in the case of PCI bursts) to / from PCI signal generation and detection.

Specifically, I’m looking for this info as it applies to Windows, XP in particular.

I’m checking into whether someone here has a membership to the PCI-SIG. Looking at their web site and the available info, I have the feeling I’m going to be needing that membership to get some of their data.

Thanks,
Judy

Just curious, why do you need to know this information, Are you designing
some custom PCI hardware that does not use a core of some sort?

wrote in message news:xxxxx@ntdev…
> Hi all,
>
> I’ve written PCI drivers for various pieces of hardware and never really
> cared how the magic happened to convert my accesses to/from the
> memory-mapped address spaces into signals on the PCI bus. I did a “data =
> *(address)” and was a happy developer. Of course, you all see where this
> is going … I now need to dig into the magic and turn it into knowledge.
> Any pointers on where I can find this info? Papers, articles, example
> source code, what pieces are in seperate drivers versus inside the kernel
> and hal versus inside the BIOS, … I’m not sure where to start looking.
> I’m familiar with how the signals should look on the actual PCI bus - I’m
> looking for the conversion from a memory-mapped access (or series of
> accesses in the case of PCI bursts) to / from PCI signal generation and
> detection.
>
> Specifically, I’m looking for this info as it applies to Windows, XP in
> particular.
>
> I’m checking into whether someone here has a membership to the PCI-SIG.
> Looking at their web site and the available info, I have the feeling I’m
> going to be needing that membership to get some of their data.
>
> Thanks,
> Judy
>
>

Possibly … We’ve got a working prototype board now, plugged into a COTS single board computer, running Windows XP. Final product (over a year down the road) will be a custom SBC board running XP Embedded connected to the custom hardware board via PCI. The end result is going to require “as fast as possible” access across the PCI bus, primarily large (on the order of megabytes) transfers in the write direction. We’re investigating all the functionality between the code in the existing prototype driver and the physical pins of the PCI bus. I’m only looking at the stuff on the SBC side of the PCI bus.

The results of the investigation will help determine how we put together the custom SBC board. We may decide to have our custom hw board be the only PCI device and therefore streamline the whole PCI process on the custom SBC and build the SBC from chipsets. Maybe that’s too complicated and we’ll use the standard PCI stuff and only modify a COTS SBC to contain our required elements. Maybe we’ll want to do the first and find a contractor who has some nitty-gritty PCI experience to help on the PCI stuff … We simply don’t know enough yet about the details to make any kind of decision. Unfortunately, no-one here has any knowledge of nor has ever dug into the “pc” side of a PCI bus, so I’m looking for any and all kinds of info.

Judy

how much data are you talking about? and what is the transfer rate you are
looking to acheive?

In our five year old implementation, we had a device that would do transfers
as large as 100MB at a time. we acheived transfer rates of close to 110MB a
sec, on the PCI bus, with large transfers and it goes down as the size goes
down(the overheads get larger). All our transfers were from the device to
the PC memory.

In our current implementation we have a PCIe device that’s currently giving
us in the order of 160MB a sec, and we’re shooting for tranfers in the order
of 200MB a sec, which should be acheivable.

In both cases, an FPGA was used to do it, ready made PCI(or PCIe) cores are
available for FPGAs which take care of the functionality on the PCI side.

hope that helps,
Ashok Bruno

wrote in message news:xxxxx@ntdev…
> Possibly … We’ve got a working prototype board now, plugged into a COTS
> single board computer, running Windows XP. Final product (over a year down
> the road) will be a custom SBC board running XP Embedded connected to the
> custom hardware board via PCI. The end result is going to require “as fast
> as possible” access across the PCI bus, primarily large (on the order of
> megabytes) transfers in the write direction. We’re investigating all the
> functionality between the code in the existing prototype driver and the
> physical pins of the PCI bus. I’m only looking at the stuff on the SBC
> side of the PCI bus.
>
> The results of the investigation will help determine how we put together
> the custom SBC board. We may decide to have our custom hw board be the
> only PCI device and therefore streamline the whole PCI process on the
> custom SBC and build the SBC from chipsets. Maybe that’s too complicated
> and we’ll use the standard PCI stuff and only modify a COTS SBC to contain
> our required elements. Maybe we’ll want to do the first and find a
> contractor who has some nitty-gritty PCI experience to help on the PCI
> stuff … We simply don’t know enough yet about the details to make any
> kind of decision. Unfortunately, no-one here has any knowledge of nor has
> ever dug into the “pc” side of a PCI bus, so I’m looking for any and all
> kinds of info.
>
> Judy
>

In some cases we’ve used products like
http://www.bcmcom.com/bcm_products.htm which might work for you, instead of
reinventing the wheel and spend development time, etc in an area where you
have little or no expertise.

wrote in message news:xxxxx@ntdev…
> Possibly … We’ve got a working prototype board now, plugged into a COTS
> single board computer, running Windows XP. Final product (over a year down
> the road) will be a custom SBC board running XP Embedded connected to the
> custom hardware board via PCI. The end result is going to require “as fast
> as possible” access across the PCI bus, primarily large (on the order of
> megabytes) transfers in the write direction. We’re investigating all the
> functionality between the code in the existing prototype driver and the
> physical pins of the PCI bus. I’m only looking at the stuff on the SBC
> side of the PCI bus.
>
> The results of the investigation will help determine how we put together
> the custom SBC board. We may decide to have our custom hw board be the
> only PCI device and therefore streamline the whole PCI process on the
> custom SBC and build the SBC from chipsets. Maybe that’s too complicated
> and we’ll use the standard PCI stuff and only modify a COTS SBC to contain
> our required elements. Maybe we’ll want to do the first and find a
> contractor who has some nitty-gritty PCI experience to help on the PCI
> stuff … We simply don’t know enough yet about the details to make any
> kind of decision. Unfortunately, no-one here has any knowledge of nor has
> ever dug into the “pc” side of a PCI bus, so I’m looking for any and all
> kinds of info.
>
> Judy
>

Thanks for the link, I’ll keep it in my pile of info. However, even if we eventually go with something like that mb, I still need to gather enough info so I can justify the decision of 3rd party versus in-house.

To answer your specific question, a typical transfer would be <1K worth of control info followed by 5MB of data from the SBC to the device.

Part of this is also my own curiousity about how this works from code to pins. The fact that we are primarily interested in transferring to the device is what has spawned the 'PC" side questions. Our hardware guys can do a custom PCI on the custom hardware card without a problem. It’s how the PCI works from within Windows as the PCI bus master that I (and they) want to know.

There was a Mindshare’s book on PCI bus.


Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

wrote in message news:xxxxx@ntdev…
> Hi all,
>
> I’ve written PCI drivers for various pieces of hardware and never really
cared how the magic happened to convert my accesses to/from the memory-mapped
address spaces into signals on the PCI bus. I did a “data = *(address)” and was
a happy developer. Of course, you all see where this is going … I now need to
dig into the magic and turn it into knowledge. Any pointers on where I can find
this info? Papers, articles, example source code, what pieces are in seperate
drivers versus inside the kernel and hal versus inside the BIOS, … I’m not
sure where to start looking. I’m familiar with how the signals should look on
the actual PCI bus - I’m looking for the conversion from a memory-mapped access
(or series of accesses in the case of PCI bursts) to / from PCI signal
generation and detection.
>
> Specifically, I’m looking for this info as it applies to Windows, XP in
particular.
>
> I’m checking into whether someone here has a membership to the PCI-SIG.
Looking at their web site and the available info, I have the feeling I’m going
to be needing that membership to get some of their data.
>
> Thanks,
> Judy
>
>

There are two levels to look at… one at the protocol layer(fortunately for
us one of the companies we bought recently makes protocol analyzers
www.lecroy.com and we use them ), where windows, etc comes in and the the
other at the actual hw layer, what the pins do, etc which you could possibly
get from PCI-SIG.

for instance when we used the protocol analyzer we were able to see that on
XP, when asked to do a Buffer write from a device driver, it was being
executed 4 bytes at a time, even though PCIe has the capability to write a
whole packet to the device.

In your case you are talking about a DMA transaction writing to the device.

In our experiments we were able to get 115MB a sec at 20MB(when transferring
from the device to the PC ofcourse assuming your device can maintain that
transfer rate), you are talking about 50 millisec to transfer the 5MB data
in a best case scenario(@ 100MB/sec). the question is : Is this enough? If
not you might consider going to PCIe or another faster bus, and then your
constraints become a little easier.

just my 2c without knowing all of your hw, sw requirements,

wrote in message news:xxxxx@ntdev…
> Thanks for the link, I’ll keep it in my pile of info. However, even if we
> eventually go with something like that mb, I still need to gather enough
> info so I can justify the decision of 3rd party versus in-house.
>
> To answer your specific question, a typical transfer would be <1K worth of
> control info followed by 5MB of data from the SBC to the device.
>
> Part of this is also my own curiousity about how this works from code to
> pins. The fact that we are primarily interested in transferring to the
> device is what has spawned the 'PC" side questions. Our hardware guys can
> do a custom PCI on the custom hardware card without a problem. It’s how
> the PCI works from within Windows as the PCI bus master that I (and
> they) want to know.
>

xxxxx@bigfoot.com wrote:

Part of this is also my own curiousity about how this works from code to pins. The fact that we are primarily interested in transferring to the device is what has spawned the 'PC" side questions. Our hardware guys can do a custom PCI on the custom hardware card without a problem. It’s how the PCI works from within Windows as the PCI bus master that I (and they) want to know.

There isn’t really that much to it. Windows doesn’t arbitrary the PCI
bus – that’s all done in hardware by the PCI controller chip. It
decides unilaterally when to grant bus master to a device, and it’s up
to the device to do the right thing when it has the bus. Windows’ only
involvement is to translate the virtual addresses to physical bus
addresses. Your driver tells your hardware what the physical addresses
are, and your hardware does the appropriate cycles after it gets the bus.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Agilent is manufacturer of PCI/PCI-X/PCI-E Analyzers and Exercisers. I
don’t know if you’re looking to spend on hardware yet or at all, but the
website (www.agilent.com) also contains links to download the
documentation, which does contain some (but not a great deal the last
time I looked) information. The two that I somewhat familiar with are
the E2920 and the E2960.

mm

>> xxxxx@hotmail.com 2007-01-10 14:03 >>>
There are two levels to look at… one at the protocol
layer(fortunately for
us one of the companies we bought recently makes protocol analyzers
www.lecroy.com and we use them ), where windows, etc comes in and the
the
other at the actual hw layer, what the pins do, etc which you could
possibly
get from PCI-SIG.

for instance when we used the protocol analyzer we were able to see
that on
XP, when asked to do a Buffer write from a device driver, it was being

executed 4 bytes at a time, even though PCIe has the capability to
write a
whole packet to the device.

In your case you are talking about a DMA transaction writing to the
device.

In our experiments we were able to get 115MB a sec at 20MB(when
transferring
from the device to the PC ofcourse assuming your device can maintain
that
transfer rate), you are talking about 50 millisec to transfer the 5MB
data
in a best case scenario(@ 100MB/sec). the question is : Is this enough?
If
not you might consider going to PCIe or another faster bus, and then
your
constraints become a little easier.

just my 2c without knowing all of your hw, sw requirements,

wrote in message news:xxxxx@ntdev…
> Thanks for the link, I’ll keep it in my pile of info. However, even
if we
> eventually go with something like that mb, I still need to gather
enough
> info so I can justify the decision of 3rd party versus in-house.
>
> To answer your specific question, a typical transfer would be <1K
worth of
> control info followed by 5MB of data from the SBC to the device.
>
> Part of this is also my own curiousity about how this works from
code to
> pins. The fact that we are primarily interested in transferring to
the
> device is what has spawned the 'PC" side questions. Our hardware guys
can
> do a custom PCI on the custom hardware card without a problem. It’s
how
> the PCI works from within Windows as the PCI bus master that I (and

> they) want to know.
>


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Tim Roberts wrote:

xxxxx@bigfoot.com wrote:

> Part of this is also my own curiousity about how this works from code to pins. The fact that we are primarily interested in transferring to the device is what has spawned the 'PC" side questions. Our hardware guys can do a custom PCI on the custom hardware card without a problem. It’s how the PCI works from within Windows as the PCI bus master that I (and they) want to know.
>
>

There isn’t really that much to it. Windows doesn’t arbitrary the PCI
bus…

Geez, did I write that? “Windows doesn’t *arbitrate* the PCI bus…”


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Neither WindowsXP nor the PCI-SIG have much to do with your question. This
is handled by your chipset. The PCI-SIG has been so profoundly successful
in part because it always steers far away from any questions about how
signals get to PCI from other busses, in particular processor busses. This
has allowed PCI to be adopted in many different environments.

Go read a chipset manual or two. This is where I get the sort of
information that you’re looking for.

  • Jake Oshins
    Windows Kernel Team

wrote in message news:xxxxx@ntdev…
> Hi all,
>
> I’ve written PCI drivers for various pieces of hardware and never really
> cared how the magic happened to convert my accesses to/from the
> memory-mapped address spaces into signals on the PCI bus. I did a “data =
> *(address)” and was a happy developer. Of course, you all see where this
> is going … I now need to dig into the magic and turn it into knowledge.
> Any pointers on where I can find this info? Papers, articles, example
> source code, what pieces are in seperate drivers versus inside the kernel
> and hal versus inside the BIOS, … I’m not sure where to start looking.
> I’m familiar with how the signals should look on the actual PCI bus - I’m
> looking for the conversion from a memory-mapped access (or series of
> accesses in the case of PCI bursts) to / from PCI signal generation and
> detection.
>
> Specifically, I’m looking for this info as it applies to Windows, XP in
> particular.
>
> I’m checking into whether someone here has a membership to the PCI-SIG.
> Looking at their web site and the available info, I have the feeling I’m
> going to be needing that membership to get some of their data.
>
> Thanks,
> Judy
>
>

There is only one small relevant, non-immediately obvious, and
Windows-specific thing that you may need to know about PCI x86 chipsets:

You want to use MmWriteCombined flag for MmMapIoSpace() in you device driver
if your PCI device protocol can handle such type of caching. It is usually
used for frame buffers but not for command registers of graphic cards.
This flag will allow the majority of current CPUs and chipsets to combine 4
double-word CPU writes to contiguous addresses to one singe 16 bytes PCI
burst transaction. I am not sure about the size of the burst (16, 32, or 64
bytes?)

Anyway, if you want to get the best performance you will need to use
bus-mastering reads and writes initiated by your PCI device and not by the
CPU / chipset. “data = *(address)” or memcpy() in the driver is not the right
way to get high performance.

Dmitriy Budko
VMware

Neither WindowsXP nor the PCI-SIG have much to do with your question.
This
is handled by your chipset. The PCI-SIG has been so profoundly successful
in part because it always steers far away from any questions about how
signals get to PCI from other busses, in particular processor busses.
This
has allowed PCI to be adopted in many different environments.

Go read a chipset manual or two. This is where I get the sort of
information that you’re looking for.

  • Jake Oshins
    Windows Kernel Team

wrote in message news:xxxxx@ntdev…
> I’m
> > looking for the conversion from a memory-mapped access (or series of
> > accesses in the case of PCI bursts) to / from PCI signal generation and
> > detection.

Thanks everyone!!! I’m off to explore chipsets, reworking our device to do bus-mastering, and other buses even though I doubt we can switch from PCI due to other issues.

Judy