Re: Flushing DMA Buffer Allocated with AllocateC ommonBuffer

I don’t see how it can work otherwise. For example, take a common-buffer DMA
where I’m writing a stream of commands to a graphics chip on the PCI bus. If
that buffer is writeback-cached, the processor is depositing the data on its
cache, and the cache lines are marked “modified” - however, there’s no data
actually written to physical memory, and it would take another front-side
bus master to access that memory to force the system to dump the modified
cache lines onto main memory. Now, the moment I kick the DMA transfer alive,
the DMA controller is going to go to memory to fetch the stream of data, the
only problem is, data isn’t there because the buffer hasn’t been flushed
from its cache lines. In fact, some of that data may not even be in the
cache yet, it may be lying in the processor’s write buffers.

But processor caches stay coherent through the MESI protocol, which operates
at hardware level and is independent of the OS. That protocol operates
through every processor snooping every transaction on the front-side bus.
But DMA controllers are masters on the PCI bus, not on the front-side bus !
Hence, I see no way a DMA controller can be aware of the comings and goings
of the MESI protocol on the front-side bus.

For example, take a look at the DDK documentation for function
VideoPortStartDma(), it specifically says that the function flushes the
memory region in the host processor’s caches, then it builds a
scatter-gather list and calls HwVidExecuteDma().

Alberto.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of Doug
Sent: Friday, November 14, 2003 11:05 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Re: Flushing DMA Buffer Allocated with
AllocateCommonBuffer

The documentation does require it. While the page in the DDK help labeled
“Using Common-Buffer Bus-Master DMA” makes no mention of allocating an MDL
and passing it to KeFlushIoBuffers, the page “Flushing Cached Data during
DMA Operations” infers it if you mark the buffers as CacheEnabled.

Problem in the DDK docs?

“Mark Roddy” wrote in message news:xxxxx@ntdev…
> 3) common buffers are ‘special’. They are platform specific designed to
> be used for dma. The documentation in the DDK does not require that you
> flush common buffers.
>
> Note that on x86 platforms KeFlushIoBuffers is a NOP. Draw your own
> conclusions.
>


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@compuware.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or disclose
it to anyone else. If you received it in error please notify us immediately
and then destroy it.

If you are writing NDIS drivers, you can use NdisMUpdateSharedMemory to
flush shared memory which is not described by any NDIS_BUFFER (MDL). Packet
descriptor structure accessed by both NIC and CPU falls into this category.

I couldn’t find the KeXxx counterpart of it. It’s defined as no-op for now
though.

Calvin Guan, Software Developer xxxxx@nospam.ati.com
SW2D-Radeon NT Core Drivers
ATI Technologies Inc.
1 Commerce Valley Drive East
Markham, Ontario, Canada L3T 7X6
Tel: (905) 882-2600 Ext. 8654
Find a driver: http://www.ati.com/support/driver.html

-----Original Message-----
From: Doug [mailto:xxxxx@hotmail.com]
Sent: Friday, November 14, 2003 11:05 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Re: Flushing DMA Buffer Allocated with
AllocateCommonBuffer

The documentation does require it. While the page in the DDK
help labeled
“Using Common-Buffer Bus-Master DMA” makes no mention of
allocating an MDL
and passing it to KeFlushIoBuffers, the page “Flushing Cached
Data during
DMA Operations” infers it if you mark the buffers as CacheEnabled.

Problem in the DDK docs?

“Mark Roddy” wrote in message
news:xxxxx@ntdev…
> 3) common buffers are ‘special’. They are platform specific designed to
> be used for dma. The documentation in the DDK does not require that you
> flush common buffers.
>
> Note that on x86 platforms KeFlushIoBuffers is a NOP. Draw your own
> conclusions.
>


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@ati.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

It’s far too slow to use uncached common buffers. Consider: for each vertex
one has to potentially issue values for x, y, z, w, s, t, and possibly color
and normals too: maybe 16 dwords per vertex. Now, think of thousands of
triangles per frame, 70 frames per second. Add textures: for example, a 1K x
1K 32-bit RGBA texture occupies 4Mb of memory. Graphics common buffers are
big and many, I’ve seen as much as 64Mb of physical memory dedicated to
graphics common buffers, and that hardly kept up with a fast 3D chip.

And have you tried to do PIO on cached memory ? You poll a status bit, and
you’re polling the cache instead - not fun !

Alberto.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of Mark Roddy
Sent: Friday, November 14, 2003 5:02 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Re: Flushing DMA Buffer Allocated with
AllocateCommonBuffer

Hmmm… I find that page more than a little confusing, don’t you? At any
rate, I do stand by my previous point 1): why is he allocating a cached
common buffer? If he would stop doing that everything would work correctly
on all platforms and he would not have to go through the idiocy of
allocating an MDL to describe a common buffer so that he can call the NOP
function KeFlushIoBuffers.

Note also that over in videoport the docs are quite different. All that the
miniport has to do is allocate the common buffer and release it. No flushing
seems to be required. Odd huh? I guess video must just be broken on NT :slight_smile:

Here is some more oddness - on the ndis side NdisUpdateSharedMemory does
nothing on any platform, and NdisFlushBuffer does nothing on current
platforms, except the IA64, where it is KeFlushIoBuffers.

Scsiport avoids this mess by forcing the miniport to use a noncached common
buffer. Hmmmm… Clue?

But if we are going to go by what the DDK docs say, check this out in the
description of KeFlushIoBuffers:

“Drivers call KeFlushIoBuffers to maintain data integrity during DMA or PIO
device transfer operations. Calling this routine affects all processors in
the machine.”

I actually have never met the driver that does a KeFlushIoBuffers every time
it finishes a PIO operation. Have any of you?

You have to call KeFlushIoBuffers on a PIO transfer for some platforms, so
by implication you have to do it on all platforms or your driver is
technically broken.

So here is what I think: x86 and amd64 snoop all memory transactions from
anywhere, and there is no issue on these platforms. IA64, like Alpha and
some of those other older platforms, has ‘issues’ here and consequently you
have to go through the idiocy of building MDLs for common buffers (or for
system buffers if you are doing PIO) so that the OS can handle edge
conditions on some platforms.

Maybe DriverFrameWorks will fix this mess?

Too bad I didn’t think to ask somebody about this at the driver lovefest in
redmond this week.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Doug
Sent: Friday, November 14, 2003 11:05 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Re: Flushing DMA Buffer Allocated with
AllocateCommonBuffer

The documentation does require it. While the page in the DDK
help labeled “Using Common-Buffer Bus-Master DMA” makes no
mention of allocating an MDL and passing it to
KeFlushIoBuffers, the page “Flushing Cached Data during DMA
Operations” infers it if you mark the buffers as CacheEnabled.

Problem in the DDK docs?

“Mark Roddy” wrote in message
> news:xxxxx@ntdev…
> > 3) common buffers are ‘special’. They are platform specific
> designed
> > to be used for dma. The documentation in the DDK does not
> require that
> > you flush common buffers.
> >
> > Note that on x86 platforms KeFlushIoBuffers is a NOP. Draw your own
> > conclusions.
> >
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as:
> xxxxx@hollistech.com To unsubscribe send a blank email to
> xxxxx@lists.osr.com
>


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@compuware.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or disclose
it to anyone else. If you received it in error please notify us immediately
and then destroy it.

The video Miniport usually only acts in setup-related chores:
initialization, mode setting, power management, retrace interrupt. It
typically does not interfere with rendering, that’s normally up to the
driver itself. So, you’re not going to find much DMA-related code in a
Miniport. For example, look at the Permedia P3 sample in the DDK, the DMA
rendering is handled in the driver and not in the Miniport. There’s not
going to be any time-intensive code inside the Miniport ! The driver must
take care of all of it, setting up the DMA, deciding on what strategy to
use, managing the live DMA environment, and writing output information to
the common buffers.

Alberto.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of Roddy, Mark
Sent: Monday, November 17, 2003 12:46 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Re: Flushing DMA Buffer Allocated with AllocateC
ommonBuffer

Video buffers tend to use write-combined caching and are special cased in
NT. As I noted earlier, the videoport interface provides no mechanism for
the miniport driver to pre or post flush a videoport common buffer.

=====================
Mark Roddy

-----Original Message-----
From: Moreira, Alberto [mailto:xxxxx@compuware.com]
Sent: Monday, November 17, 2003 10:30 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Re: Flushing DMA Buffer Allocated with
AllocateC ommonBuffer

It’s far too slow to use uncached common buffers. Consider:
for each vertex one has to potentially issue values for x, y,
z, w, s, t, and possibly color and normals too: maybe 16
dwords per vertex. Now, think of thousands of triangles per
frame, 70 frames per second. Add textures: for example, a 1K
x 1K 32-bit RGBA texture occupies 4Mb of memory. Graphics
common buffers are big and many, I’ve seen as much as 64Mb of
physical memory dedicated to graphics common buffers, and
that hardly kept up with a fast 3D chip.

And have you tried to do PIO on cached memory ? You poll a
status bit, and you’re polling the cache instead - not fun !

Alberto.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of Mark Roddy
Sent: Friday, November 14, 2003 5:02 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Re: Flushing DMA Buffer Allocated with
AllocateCommonBuffer

Hmmm… I find that page more than a little confusing, don’t
you? At any rate, I do stand by my previous point 1): why is
he allocating a cached common buffer? If he would stop doing
that everything would work correctly on all platforms and he
would not have to go through the idiocy of allocating an MDL
to describe a common buffer so that he can call the NOP
function KeFlushIoBuffers.

Note also that over in videoport the docs are quite
different. All that the miniport has to do is allocate the
common buffer and release it. No flushing seems to be
required. Odd huh? I guess video must just be broken on NT :slight_smile:

Here is some more oddness - on the ndis side
NdisUpdateSharedMemory does nothing on any platform, and
NdisFlushBuffer does nothing on current platforms, except the
IA64, where it is KeFlushIoBuffers.

Scsiport avoids this mess by forcing the miniport to use a
noncached common buffer. Hmmmm… Clue?

But if we are going to go by what the DDK docs say, check
this out in the description of KeFlushIoBuffers:

“Drivers call KeFlushIoBuffers to maintain data integrity
during DMA or PIO device transfer operations. Calling this
routine affects all processors in the machine.”

I actually have never met the driver that does a
KeFlushIoBuffers every time it finishes a PIO operation. Have
any of you?

You have to call KeFlushIoBuffers on a PIO transfer for some
platforms, so by implication you have to do it on all
platforms or your driver is technically broken.

So here is what I think: x86 and amd64 snoop all memory
transactions from anywhere, and there is no issue on these
platforms. IA64, like Alpha and some of those other older
platforms, has ‘issues’ here and consequently you have to go
through the idiocy of building MDLs for common buffers (or
for system buffers if you are doing PIO) so that the OS can
handle edge conditions on some platforms.

Maybe DriverFrameWorks will fix this mess?

Too bad I didn’t think to ask somebody about this at the
driver lovefest in redmond this week.

> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Doug
> Sent: Friday, November 14, 2003 11:05 AM
> To: Windows System Software Devs Interest List
> Subject: [ntdev] Re: Flushing DMA Buffer Allocated with
> AllocateCommonBuffer
>
> The documentation does require it. While the page in the DDK help
> labeled “Using Common-Buffer Bus-Master DMA” makes no mention of
> allocating an MDL and passing it to KeFlushIoBuffers, the page
> “Flushing Cached Data during DMA Operations” infers it if
you mark the
> buffers as CacheEnabled.
>
> Problem in the DDK docs?
>
> “Mark Roddy” wrote in message
> > news:xxxxx@ntdev…
> > > 3) common buffers are ‘special’. They are platform specific
> > designed
> > > to be used for dma. The documentation in the DDK does not
> > require that
> > > you flush common buffers.
> > >
> > > Note that on x86 platforms KeFlushIoBuffers is a NOP.
> Draw your own
> > > conclusions.
> > >
> >
> >
> >
> > —
> > Questions? First check the Kernel Driver FAQ at
> > http://www.osronline.com/article.cfm?id=256
> >
> > You are currently subscribed to ntdev as:
> > xxxxx@hollistech.com To unsubscribe send a blank email to
> > xxxxx@lists.osr.com
> >
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as:
> xxxxx@compuware.com To unsubscribe send a blank
> email to xxxxx@lists.osr.com
>
>
>
> The contents of this e-mail are intended for the named
> addressee only. It contains information that may be
> confidential. Unless you are the named addressee or an
> authorized designee, you may not copy or use it, or disclose
> it to anyone else. If you received it in error please notify
> us immediately and then destroy it.
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as:
> xxxxx@stratus.com To unsubscribe send a blank email to
> xxxxx@lists.osr.com
>


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@compuware.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or disclose
it to anyone else. If you received it in error please notify us immediately
and then destroy it.