Re: posible bug in XP kernel - was MmMapLockedPages.. . on XP vs. 2000

I’m debating with myself here :slight_smile:

We thought about the cached writer problem (the non-cached reader could
get stale, thus incoherent data,) but we came up with this
falsification: pci devices doing dma will fetch data from cpu cache’s if
required rather than physical memory due to the intervention of the
memory bus chipset. Bus snooping will prevent either cached writers or
non-cached writers from presenting an out of order sequence to memory
readers.

So we continue to be puzzled about exactly what bug the change in XP mm
code was fixing. Perhaps it is fixing a bug that does not exist on
current pentium* architecture platforms?

-----Original Message-----
From: Roddy, Mark [mailto:xxxxx@stratus.com]
Sent: Friday, June 21, 2002 12:22 PM
To: NT Developers Interest List
Subject: [ntdev] Re: posible bug in XP kernel - was
MmMapLockedPages… . on XP vs. 2000

Actually I think it is the cached writers who are the
problem, not the non-cached writers. The non-cached write to
memory will indeed invalidate the cache lines of the cached
writer cpus, due to memory bus snooping, however the cached
writers will not invalidate the cache lines of the non-cached
writer, 'cause he ain’t using the cache. The cpus using cache
will have a coherent view, the cpu(s) doing non-cached access
may not. Particularly a problem if invalidates can be
statisfied from cache rather than from memory.

> -----Original Message-----
> From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
> Sent: Friday, June 21, 2002 10:53 AM
> To: NT Developers Interest List
> Subject: [ntdev] Re: posible bug in XP kernel - was
> MmMapLockedPages… . on XP vs. 2000
>
>
> > everywhere, but can somebody explain exactly how the cache
> coherency
> > protocol gets violated in this case and memory access gets
> potentially
> > incoherent?
>
> Imagine the same page double-mapped to the address space by 2
> different PTEs, one with caching on, other with caching off.
> Now imagine some code writes a value via noncached PTE. The
> value is updated in memory and not updated in the cache. The
> read from cached PTE will read the stale value.
>
> This is a good old problem which is known to people working
> on hardware which requires the software to know on cache
> coherency issues (i.e. PowerPC).
>
> Max
>
>
>
> —
> You are currently subscribed to ntdev as:
> xxxxx@stratus.com To unsubscribe send a blank email to
> %%email.unsub%%
>


You are currently subscribed to ntdev as: xxxxx@stratus.com
To unsubscribe send a blank email to %%email.unsub%%

Mark,

Im debating with myself too, and Intel architecture manuals:

check Intel Architecture Software Developer’s Manual Volume 3: System
Programming Guide

part number 24319202

Section 9.13.5 Programming the PAT

Now, this seems very consistent , and folowed by XP kernel. The guidelines
are folowed, as well they ensure corectly that any PAT entry is validate by
flushing the TLB and caches are invalidated by sending an IPI to all
processors, which will basically just execute a wbinvd. (check MmMapIoSpace
in debugger) This is done only for p6+ . I think the problem is WITH actual
processors and not older one.

I wait for your feedback after you read this

----- Original Message -----
From: “Mark Roddy”
To: “NT Developers Interest List”
Sent: Saturday, June 22, 2002 6:52 PM
Subject: [ntdev] Re: posible bug in XP kernel - was MmMapLockedPages… . on
XP vs. 2000

> I’m debating with myself here :slight_smile:
>
> We thought about the cached writer problem (the non-cached reader could
> get stale, thus incoherent data,) but we came up with this
> falsification: pci devices doing dma will fetch data from cpu cache’s if
> required rather than physical memory due to the intervention of the
> memory bus chipset. Bus snooping will prevent either cached writers or
> non-cached writers from presenting an out of order sequence to memory
> readers.
>
> So we continue to be puzzled about exactly what bug the change in XP mm
> code was fixing. Perhaps it is fixing a bug that does not exist on
> current pentium* architecture platforms?
>
> > -----Original Message-----
> > From: Roddy, Mark [mailto:xxxxx@stratus.com]
> > Sent: Friday, June 21, 2002 12:22 PM
> > To: NT Developers Interest List
> > Subject: [ntdev] Re: posible bug in XP kernel - was
> > MmMapLockedPages… . on XP vs. 2000
> >
> >
> > Actually I think it is the cached writers who are the
> > problem, not the non-cached writers. The non-cached write to
> > memory will indeed invalidate the cache lines of the cached
> > writer cpus, due to memory bus snooping, however the cached
> > writers will not invalidate the cache lines of the non-cached
> > writer, 'cause he ain’t using the cache. The cpus using cache
> > will have a coherent view, the cpu(s) doing non-cached access
> > may not. Particularly a problem if invalidates can be
> > statisfied from cache rather than from memory.
> >
> > > -----Original Message-----
> > > From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
> > > Sent: Friday, June 21, 2002 10:53 AM
> > > To: NT Developers Interest List
> > > Subject: [ntdev] Re: posible bug in XP kernel - was
> > > MmMapLockedPages… . on XP vs. 2000
> > >
> > >
> > > > everywhere, but can somebody explain exactly how the cache
> > > coherency
> > > > protocol gets violated in this case and memory access gets
> > > potentially
> > > > incoherent?
> > >
> > > Imagine the same page double-mapped to the address space by 2
> > > different PTEs, one with caching on, other with caching off.
> > > Now imagine some code writes a value via noncached PTE. The
> > > value is updated in memory and not updated in the cache. The
> > > read from cached PTE will read the stale value.
> > >
> > > This is a good old problem which is known to people working
> > > on hardware which requires the software to know on cache
> > > coherency issues (i.e. PowerPC).
> > >
> > > Max
> > >
> > >
> > >
> > > —
> > > You are currently subscribed to ntdev as:
> > > xxxxx@stratus.com To unsubscribe send a blank email to
> > > %%email.unsub%%
> > >
> >
> > —
> > You are currently subscribed to ntdev as: xxxxx@stratus.com
> > To unsubscribe send a blank email to %%email.unsub%%
> >
>
>
>
> —
> You are currently subscribed to ntdev as: xxxxx@rdsor.ro
> To unsubscribe send a blank email to %%email.unsub%%
>

Well I continue to debate with myself. My reading of the Intel manual
was interpretive: they actual discourage any divergence in usage. I
still wonder why. The one case the document expressly forbids is the WC
case, where access to the cache is forbidden, consequently no processor
can map the pages in question cached. I remain puzzled about the
incoherency that could result from the non-cached vs cached access case.
At any rate, it is of interest only to folks doing stupid memory tricks,
and how many of them are there?

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Dan Partelly
Sent: Saturday, June 22, 2002 12:18 PM
To: NT Developers Interest List
Subject: [ntdev] Re: posible bug in XP kernel - was
MmMapLockedPages… . on XP vs. 2000

Mark,

Im debating with myself too, and Intel architecture manuals:

check Intel Architecture Software Developer’s Manual Volume
3: System Programming Guide

part number 24319202

Section 9.13.5 Programming the PAT

Now, this seems very consistent , and folowed by XP kernel.
The guidelines are folowed, as well they ensure corectly that
any PAT entry is validate by flushing the TLB and caches are
invalidated by sending an IPI to all processors, which will
basically just execute a wbinvd. (check MmMapIoSpace in
debugger) This is done only for p6+ . I think the problem is
WITH actual processors and not older one.

I wait for your feedback after you read this

----- Original Message -----
From: “Mark Roddy”
> To: “NT Developers Interest List”
> Sent: Saturday, June 22, 2002 6:52 PM
> Subject: [ntdev] Re: posible bug in XP kernel - was
> MmMapLockedPages… . on XP vs. 2000
>
>
> > I’m debating with myself here :slight_smile:
> >
> > We thought about the cached writer problem (the non-cached reader
> > could get stale, thus incoherent data,) but we came up with this
> > falsification: pci devices doing dma will fetch data from
> cpu cache’s
> > if required rather than physical memory due to the
> intervention of the
> > memory bus chipset. Bus snooping will prevent either cached
> writers or
> > non-cached writers from presenting an out of order sequence
> to memory
> > readers.
> >
> > So we continue to be puzzled about exactly what bug the
> change in XP
> > mm code was fixing. Perhaps it is fixing a bug that does
> not exist on
> > current pentium* architecture platforms?
> >
> > > -----Original Message-----
> > > From: Roddy, Mark [mailto:xxxxx@stratus.com]
> > > Sent: Friday, June 21, 2002 12:22 PM
> > > To: NT Developers Interest List
> > > Subject: [ntdev] Re: posible bug in XP kernel - was
> > > MmMapLockedPages… . on XP vs. 2000
> > >
> > >
> > > Actually I think it is the cached writers who are the
> problem, not
> > > the non-cached writers. The non-cached write to memory
> will indeed
> > > invalidate the cache lines of the cached writer cpus, due
> to memory
> > > bus snooping, however the cached writers will not invalidate the
> > > cache lines of the non-cached writer, 'cause he ain’t using the
> > > cache. The cpus using cache will have a coherent view, the cpu(s)
> > > doing non-cached access may not. Particularly a problem if
> > > invalidates can be statisfied from cache rather than from memory.
> > >
> > > > -----Original Message-----
> > > > From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
> > > > Sent: Friday, June 21, 2002 10:53 AM
> > > > To: NT Developers Interest List
> > > > Subject: [ntdev] Re: posible bug in XP kernel - was
> > > > MmMapLockedPages… . on XP vs. 2000
> > > >
> > > >
> > > > > everywhere, but can somebody explain exactly how the cache
> > > > coherency
> > > > > protocol gets violated in this case and memory access gets
> > > > potentially
> > > > > incoherent?
> > > >
> > > > Imagine the same page double-mapped to the address space by 2
> > > > different PTEs, one with caching on, other with caching
> off. Now
> > > > imagine some code writes a value via noncached PTE. The
> value is
> > > > updated in memory and not updated in the cache. The read from
> > > > cached PTE will read the stale value.
> > > >
> > > > This is a good old problem which is known to people working on
> > > > hardware which requires the software to know on cache coherency
> > > > issues (i.e. PowerPC).
> > > >
> > > > Max
> > > >
> > > >
> > > >
> > > > —
> > > > You are currently subscribed to ntdev as:
> xxxxx@stratus.com
> > > > To unsubscribe send a blank email to %%email.unsub%%
> > > >
> > >
> > > —
> > > You are currently subscribed to ntdev as:
> xxxxx@stratus.com To
> > > unsubscribe send a blank email to %%email.unsub%%
> > >
> >
> >
> >
> > —
> > You are currently subscribed to ntdev as: xxxxx@rdsor.ro To
> > unsubscribe send a blank email to %%email.unsub%%
> >
>
>
> —
> You are currently subscribed to ntdev as:
> xxxxx@hollistech.com To unsubscribe send a blank email to
> %%email.unsub%%
>
>

> At any rate, it is of interest only to folks doing stupid memory tricks,

and how many of them are there?

Considering the great feedback we get in this thread, Id say people interes
in this is like 0, Nothing, Nada…

Ill check Intel chipset manuals, maybe some new clues will reveal from
there. I do agree with your interpreation, Intel has problem with WC vs WB
Cached memory, even if they discourage all conflicting cases, they only
specifically talk about this one. Anyway, maybe we should check AMD CPU
manuals too.

Anyone to throw more clues for us ? MS ppl, no feedback at all ?

Dan

----- Original Message -----
From: “Mark Roddy”
To: “NT Developers Interest List”
Sent: Thursday, June 27, 2002 5:01 AM
Subject: [ntdev] Re: posible bug in XP kernel - was MmMapLockedPages… . on
XP vs. 2000

> Well I continue to debate with myself. My reading of the Intel manual
> was interpretive: they actual discourage any divergence in usage. I
> still wonder why. The one case the document expressly forbids is the WC
> case, where access to the cache is forbidden, consequently no processor
> can map the pages in question cached. I remain puzzled about the
> incoherency that could result from the non-cached vs cached access case.
> At any rate, it is of interest only to folks doing stupid memory tricks,
> and how many of them are there?
>
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of Dan Partelly
> > Sent: Saturday, June 22, 2002 12:18 PM
> > To: NT Developers Interest List
> > Subject: [ntdev] Re: posible bug in XP kernel - was
> > MmMapLockedPages… . on XP vs. 2000
> >
> >
> > Mark,
> >
> > Im debating with myself too, and Intel architecture manuals:
> >
> > check Intel Architecture Software Developer’s Manual Volume
> > 3: System Programming Guide
> >
> > part number 24319202
> >
> > Section 9.13.5 Programming the PAT
> >
> > Now, this seems very consistent , and folowed by XP kernel.
> > The guidelines are folowed, as well they ensure corectly that
> > any PAT entry is validate by flushing the TLB and caches are
> > invalidated by sending an IPI to all processors, which will
> > basically just execute a wbinvd. (check MmMapIoSpace in
> > debugger) This is done only for p6+ . I think the problem is
> > WITH actual processors and not older one.
> >
> > I wait for your feedback after you read this
> >
> > ----- Original Message -----
> > From: “Mark Roddy”
> > To: “NT Developers Interest List”
> > Sent: Saturday, June 22, 2002 6:52 PM
> > Subject: [ntdev] Re: posible bug in XP kernel - was
> > MmMapLockedPages… . on XP vs. 2000
> >
> >
> > > I’m debating with myself here :slight_smile:
> > >
> > > We thought about the cached writer problem (the non-cached reader
> > > could get stale, thus incoherent data,) but we came up with this
> > > falsification: pci devices doing dma will fetch data from
> > cpu cache’s
> > > if required rather than physical memory due to the
> > intervention of the
> > > memory bus chipset. Bus snooping will prevent either cached
> > writers or
> > > non-cached writers from presenting an out of order sequence
> > to memory
> > > readers.
> > >
> > > So we continue to be puzzled about exactly what bug the
> > change in XP
> > > mm code was fixing. Perhaps it is fixing a bug that does
> > not exist on
> > > current pentium* architecture platforms?
> > >
> > > > -----Original Message-----
> > > > From: Roddy, Mark [mailto:xxxxx@stratus.com]
> > > > Sent: Friday, June 21, 2002 12:22 PM
> > > > To: NT Developers Interest List
> > > > Subject: [ntdev] Re: posible bug in XP kernel - was
> > > > MmMapLockedPages… . on XP vs. 2000
> > > >
> > > >
> > > > Actually I think it is the cached writers who are the
> > problem, not
> > > > the non-cached writers. The non-cached write to memory
> > will indeed
> > > > invalidate the cache lines of the cached writer cpus, due
> > to memory
> > > > bus snooping, however the cached writers will not invalidate the
> > > > cache lines of the non-cached writer, 'cause he ain’t using the
> > > > cache. The cpus using cache will have a coherent view, the cpu(s)
> > > > doing non-cached access may not. Particularly a problem if
> > > > invalidates can be statisfied from cache rather than from memory.
> > > >
> > > > > -----Original Message-----
> > > > > From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
> > > > > Sent: Friday, June 21, 2002 10:53 AM
> > > > > To: NT Developers Interest List
> > > > > Subject: [ntdev] Re: posible bug in XP kernel - was
> > > > > MmMapLockedPages… . on XP vs. 2000
> > > > >
> > > > >
> > > > > > everywhere, but can somebody explain exactly how the cache
> > > > > coherency
> > > > > > protocol gets violated in this case and memory access gets
> > > > > potentially
> > > > > > incoherent?
> > > > >
> > > > > Imagine the same page double-mapped to the address space by 2
> > > > > different PTEs, one with caching on, other with caching
> > off. Now
> > > > > imagine some code writes a value via noncached PTE. The
> > value is
> > > > > updated in memory and not updated in the cache. The read from
> > > > > cached PTE will read the stale value.
> > > > >
> > > > > This is a good old problem which is known to people working on
> > > > > hardware which requires the software to know on cache coherency
> > > > > issues (i.e. PowerPC).
> > > > >
> > > > > Max
> > > > >
> > > > >
> > > > >
> > > > > —
> > > > > You are currently subscribed to ntdev as:
> > xxxxx@stratus.com
> > > > > To unsubscribe send a blank email to %%email.unsub%%
> > > > >
> > > >
> > > > —
> > > > You are currently subscribed to ntdev as:
> > xxxxx@stratus.com To
> > > > unsubscribe send a blank email to %%email.unsub%%
> > > >
> > >
> > >
> > >
> > > —
> > > You are currently subscribed to ntdev as: xxxxx@rdsor.ro To
> > > unsubscribe send a blank email to %%email.unsub%%
> > >
> >
> >
> > —
> > You are currently subscribed to ntdev as:
> > xxxxx@hollistech.com To unsubscribe send a blank email to
> > %%email.unsub%%
> >
> >
>
>
>
> —
> You are currently subscribed to ntdev as: xxxxx@rdsor.ro
> To unsubscribe send a blank email to %%email.unsub%%
>