Terminating of file caching

NTFSD
1

Happy New Year!!!

I am trying to write the simple network redirector for W2K and I have
a few problems with terminating of file caching.

My problems concerning the interaction between IRP_MJ_READ,
IRP_MJ_CLEANUP and IRP_MJ_CLOSE requests.
When an test application closes the file (i.e. _close()), my driver
gets the IRP_MJ_CLEANUP. If I use the caching of file ( i.e. I use
CcInitializeCacheMap(), CcSetReadAheadGranularity() and CcCopyRead()
in read() subroutine and CcFlushCache(), CcPurgeCache() and
CcUninitializeCacheMap() in cleanup() subroutine) the FSD doesn’t receive
the IRP_MJ_CLOSE request for this file. It is first problem.

The second problem: The CcUninitializeCacheMap() always return FALSE
for caching and non caching files. Is it correct?

The third problem: in 1 - 2 seconds after the completion of IPR_MJ_CLEANUP
request for caching file, the system breaks in DbgBreakPoint in EXWORKER
and I see the follow messages:

>EXWORKER: exception in worker routine 80414C70(FD90D6A8)
> exception record at F1437BC4
> context record at F143781C
>ntoskrnl!DbgBreakPoint:
>804a3ce4 cc int 3
>kd> kb
>
>ChildEBP RetAddr Args to Child
>f14376e0 8041c1c9 80401020 00000000 f1437d98 ntoskrnl!DbgBreakPoint
>f143770c 8041c0f3 80414c70 fd90d6a8 f143773c
ntoskrnl!ExpWorkerThreadFilter+0x61
>f1437da8 804a331a 00000000 00000000 00000000
ntoskrnl!ExpWorkerThread+0x191
>f1437ddc 804bdd7a 8041bf62 00000000 00000000
ntoskrnl!PspSystemThreadStartup+0x54
>00000000 00000000 00000000 00000000 00000000 ntoskrnl!KiThreadStartup+0x16
>

To continue I must press twice the ‘g’ key.

The confusion is that if I delete CcUninitializeCacheMap() from
cleanup() subroutine, I have not this problem.

My questions:

  1. What are the reasons of this problems?
  2. Where can I find the correct sequence of using the Cache Manager
    and Memory Manager interface for full terminating of file caching?

I found in ntfsd archive that similar problem was discussed by Mark Twain,
Maxim S. Shatskih, Jack Brown and Inaki Castillo at April 2000.
What are their opinions?

Thanks in advance.

Nikolay Naumov
E-mail: xxxxx@mikron.ru
Mikron Corporation, http://www.mikron.ru
Moscow, Russia

You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

2

Your caching logic is incorrect.

IRP_MJ_CLEANUP merely denotes when the USER is done with the file object.
It has nothing to do with the ongoing use of the file object by the OS.

IRP_MJ_CLOSE indicates that all use of the file object will cease. For the
file object that is used as the backing file object for the cache, the time
between CLEANUP and CLOSE can be very long.

That it is crashing is indictative of a bug in your code or logic. You are
not handling some background processing case properly. You need to debug
this. If you review the documentation for bug check code 0x1E it will tell
you how to determine where the exception occurred in your code so that you
can figure out where the bug is. You do this by looking at the exception
record and context record that are stored in memory.

Regards,

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: Naumov [mailto:xxxxx@Mikron.RU]
Sent: Wednesday, December 27, 2000 10:01 AM
To: File Systems Developers
Subject: [ntfsd] Terminating of file caching

Happy New Year!!!

I am trying to write the simple network redirector for W2K and I have
a few problems with terminating of file caching.

My problems concerning the interaction between IRP_MJ_READ,
IRP_MJ_CLEANUP and IRP_MJ_CLOSE requests.
When an test application closes the file (i.e. _close()), my driver
gets the IRP_MJ_CLEANUP. If I use the caching of file ( i.e. I use
CcInitializeCacheMap(), CcSetReadAheadGranularity() and CcCopyRead()
in read() subroutine and CcFlushCache(), CcPurgeCache() and
CcUninitializeCacheMap() in cleanup() subroutine) the FSD doesn’t receive
the IRP_MJ_CLOSE request for this file. It is first problem.

The second problem: The CcUninitializeCacheMap() always return FALSE
for caching and non caching files. Is it correct?

The third problem: in 1 - 2 seconds after the completion of IPR_MJ_CLEANUP
request for caching file, the system breaks in DbgBreakPoint in EXWORKER
and I see the follow messages:

>EXWORKER: exception in worker routine 80414C70(FD90D6A8)
> exception record at F1437BC4
> context record at F143781C
>ntoskrnl!DbgBreakPoint:
>804a3ce4 cc int 3
>kd> kb
>
>ChildEBP RetAddr Args to Child
>f14376e0 8041c1c9 80401020 00000000 f1437d98 ntoskrnl!DbgBreakPoint
>f143770c 8041c0f3 80414c70 fd90d6a8 f143773c
ntoskrnl!ExpWorkerThreadFilter+0x61
>f1437da8 804a331a 00000000 00000000 00000000
ntoskrnl!ExpWorkerThread+0x191
>f1437ddc 804bdd7a 8041bf62 00000000 00000000
ntoskrnl!PspSystemThreadStartup+0x54
>00000000 00000000 00000000 00000000 00000000 ntoskrnl!KiThreadStartup+0x16
>

To continue I must press twice the ‘g’ key.

The confusion is that if I delete CcUninitializeCacheMap() from
cleanup() subroutine, I have not this problem.

My questions:

  1. What are the reasons of this problems?
  2. Where can I find the correct sequence of using the Cache Manager
    and Memory Manager interface for full terminating of file caching?

I found in ntfsd archive that similar problem was discussed by Mark Twain,
Maxim S. Shatskih, Jack Brown and Inaki Castillo at April 2000.
What are their opinions?

Thanks in advance.

Nikolay Naumov
E-mail: xxxxx@mikron.ru
Mikron Corporation, http://www.mikron.ru
Moscow, Russia

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

3

specifically you can do the following to help debug this:

!exr F1437BC4
!cxr F143781C
!kb

the hex values are from the bugcheck information.

-----Original Message-----
From: Tony Mason [mailto:xxxxx@osr.com]
Sent: Wednesday, December 27, 2000 1:04 PM
To: File Systems Developers
Subject: [ntfsd] RE: Terminating of file caching

Your caching logic is incorrect.

IRP_MJ_CLEANUP merely denotes when the USER is done with the file object.
It has nothing to do with the ongoing use of the file object by the OS.

IRP_MJ_CLOSE indicates that all use of the file object will cease. For the
file object that is used as the backing file object for the cache, the time
between CLEANUP and CLOSE can be very long.

That it is crashing is indictative of a bug in your code or logic. You are
not handling some background processing case properly. You need to debug
this. If you review the documentation for bug check code 0x1E it will tell
you how to determine where the exception occurred in your code so that you
can figure out where the bug is. You do this by looking at the exception
record and context record that are stored in memory.

Regards,

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: Naumov [mailto:xxxxx@Mikron.RU]
Sent: Wednesday, December 27, 2000 10:01 AM
To: File Systems Developers
Subject: [ntfsd] Terminating of file caching

Happy New Year!!!

I am trying to write the simple network redirector for W2K and I have
a few problems with terminating of file caching.

My problems concerning the interaction between IRP_MJ_READ,
IRP_MJ_CLEANUP and IRP_MJ_CLOSE requests.
When an test application closes the file (i.e. _close()), my driver
gets the IRP_MJ_CLEANUP. If I use the caching of file ( i.e. I use
CcInitializeCacheMap(), CcSetReadAheadGranularity() and CcCopyRead()
in read() subroutine and CcFlushCache(), CcPurgeCache() and
CcUninitializeCacheMap() in cleanup() subroutine) the FSD doesn’t receive
the IRP_MJ_CLOSE request for this file. It is first problem.

The second problem: The CcUninitializeCacheMap() always return FALSE
for caching and non caching files. Is it correct?

The third problem: in 1 - 2 seconds after the completion of IPR_MJ_CLEANUP
request for caching file, the system breaks in DbgBreakPoint in EXWORKER
and I see the follow messages:

>EXWORKER: exception in worker routine 80414C70(FD90D6A8)
> exception record at F1437BC4
> context record at F143781C
>ntoskrnl!DbgBreakPoint:
>804a3ce4 cc int 3
>kd> kb
>
>ChildEBP RetAddr Args to Child
>f14376e0 8041c1c9 80401020 00000000 f1437d98 ntoskrnl!DbgBreakPoint
>f143770c 8041c0f3 80414c70 fd90d6a8 f143773c
ntoskrnl!ExpWorkerThreadFilter+0x61
>f1437da8 804a331a 00000000 00000000 00000000
ntoskrnl!ExpWorkerThread+0x191
>f1437ddc 804bdd7a 8041bf62 00000000 00000000
ntoskrnl!PspSystemThreadStartup+0x54
>00000000 00000000 00000000 00000000 00000000 ntoskrnl!KiThreadStartup+0x16
>

To continue I must press twice the ‘g’ key.

The confusion is that if I delete CcUninitializeCacheMap() from
cleanup() subroutine, I have not this problem.

My questions:

  1. What are the reasons of this problems?
  2. Where can I find the correct sequence of using the Cache Manager
    and Memory Manager interface for full terminating of file caching?

I found in ntfsd archive that similar problem was discussed by Mark Twain,
Maxim S. Shatskih, Jack Brown and Inaki Castillo at April 2000.
What are their opinions?

Thanks in advance.

Nikolay Naumov
E-mail: xxxxx@mikron.ru
Mikron Corporation, http://www.mikron.ru
Moscow, Russia

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntfsd as: xxxxx@veritas.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

4

Tony Mason and Wesley Witt,

Thanks very much for Your replays.
They help me to fix bug in my code.

But I do not understand the way to purge cache and
re-read the cached file from remote server.
Is it enough to use the CcPurgeCache()?
Can I set any “time of life” of cached data, and
if this time is up, the system preform the request
to FSD to re-read the cached file?

Thanks in advance.

Nikolay Naumov
E-mail: xxxxx@mikron.ru
Mikron Corporation, http://www.mikron.ru
Moscow, Russia

----- Original Message -----
From: Wesley Witt
To: File Systems Developers
Sent: Thursday, December 28, 2000 1:10 AM
Subject: [ntfsd] RE: Terminating of file caching

> specifically you can do the following to help debug this:
>
> !exr F1437BC4
> !cxr F143781C
> !kb
>
> the hex values are from the bugcheck information.
>
> -----Original Message-----
> From: Tony Mason [mailto:xxxxx@osr.com]
> Sent: Wednesday, December 27, 2000 1:04 PM
> To: File Systems Developers
> Subject: [ntfsd] RE: Terminating of file caching
>
>
> Your caching logic is incorrect.
>
> IRP_MJ_CLEANUP merely denotes when the USER is done with the file object.
> It has nothing to do with the ongoing use of the file object by the OS.
>
> IRP_MJ_CLOSE indicates that all use of the file object will cease. For
the
> file object that is used as the backing file object for the cache, the
time
> between CLEANUP and CLOSE can be very long.
>
> That it is crashing is indictative of a bug in your code or logic. You
are
> not handling some background processing case properly. You need to debug
> this. If you review the documentation for bug check code 0x1E it will
tell
> you how to determine where the exception occurred in your code so that you
> can figure out where the bug is. You do this by looking at the exception
> record and context record that are stored in memory.
>
> Regards,
>
> Tony Mason
> Consulting Partner
> OSR Open Systems Resources, Inc.
> http://www.osr.com
>
> …
Skipped…
>

—
You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

5

> My problems concerning the interaction between IRP_MJ_READ,

IRP_MJ_CLEANUP and IRP_MJ_CLOSE requests.
When an test application closes the file (i.e. _close()), my driver
gets the IRP_MJ_CLEANUP. If I use the caching of file ( i.e. I use
CcInitializeCacheMap(), CcSetReadAheadGranularity() and CcCopyRead()
in read() subroutine and CcFlushCache(), CcPurgeCache() and
CcUninitializeCacheMap() in cleanup() subroutine) the FSD doesn’t receive
the IRP_MJ_CLOSE request for this file. It is first problem.

You will not get CLOSE for the file till Cc is holding a reference to it.

On CLEANUP, try to do:

  • CcFlushCache
    then
  • MmFlushImageSection
  • CcPurgeCacheSection
  • CcUninitializeCacheMap

The cache map for the file will be placed to the “dead” list by the Cc, Cc
will then delete the cache map and dereference the file object, thus causing
CLOSE to be sent.
Usually it will occur rather swiftly - provided you have called all of the
above calls on CLEANUP.

Max

You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

6

Hello, Max

Thanks for Your replay.
It seems to me that the file caching is working now.
Happy New Year!!!

Nikolay Naumov
E-mail: xxxxx@mikron.ru
Mikron Corporation, http://www.mikron.ru
Moscow, Russia

----- Original Message -----
From: Maxim S. Shatskih
To: File Systems Developers
Sent: Thursday, December 28, 2000 11:19 PM
Subject: [ntfsd] Re: Terminating of file caching

> > My problems concerning the interaction between IRP_MJ_READ,
> > IRP_MJ_CLEANUP and IRP_MJ_CLOSE requests.
> > When an test application closes the file (i.e. _close()), my driver
> > gets the IRP_MJ_CLEANUP. If I use the caching of file ( i.e. I use
> > CcInitializeCacheMap(), CcSetReadAheadGranularity() and CcCopyRead()
> > in read() subroutine and CcFlushCache(), CcPurgeCache() and
> > CcUninitializeCacheMap() in cleanup() subroutine) the FSD doesn’t
receive
> > the IRP_MJ_CLOSE request for this file. It is first problem.
>
> You will not get CLOSE for the file till Cc is holding a reference to it.
>
> On CLEANUP, try to do:
> - CcFlushCache
> then
> - MmFlushImageSection
> - CcPurgeCacheSection
> - CcUninitializeCacheMap
>
> The cache map for the file will be placed to the “dead” list by the Cc, Cc
> will then delete the cache map and dereference the file object, thus
causing
> CLOSE to be sent.
> Usually it will occur rather swiftly - provided you have called all of the
> above calls on CLEANUP.
>
> Max
>
>
> —
> You are currently subscribed to ntfsd as: xxxxx@Mikron.RU
> To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com
>

—
You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

7

Happy new year for you too!

The greatest Russian joke of the day:

http://forum.mts.ru

:-)))

Max

----- Original Message -----
From: “Naumov”
To: “File Systems Developers”
Sent: Friday, December 29, 2000 9:29 AM
Subject: [ntfsd] Re: Terminating of file caching

> Hello, Max
>
> Thanks for Your replay.
> It seems to me that the file caching is working now.
> Happy New Year!!!
>
> Nikolay Naumov
> E-mail: xxxxx@mikron.ru
> Mikron Corporation, http://www.mikron.ru
> Moscow, Russia
>
>
> ----- Original Message -----
> From: Maxim S. Shatskih
> To: File Systems Developers
> Sent: Thursday, December 28, 2000 11:19 PM
> Subject: [ntfsd] Re: Terminating of file caching
>
>
> > > My problems concerning the interaction between IRP_MJ_READ,
> > > IRP_MJ_CLEANUP and IRP_MJ_CLOSE requests.
> > > When an test application closes the file (i.e. _close()), my driver
> > > gets the IRP_MJ_CLEANUP. If I use the caching of file ( i.e. I use
> > > CcInitializeCacheMap(), CcSetReadAheadGranularity() and CcCopyRead()
> > > in read() subroutine and CcFlushCache(), CcPurgeCache() and
> > > CcUninitializeCacheMap() in cleanup() subroutine) the FSD doesn’t
> receive
> > > the IRP_MJ_CLOSE request for this file. It is first problem.
> >
> > You will not get CLOSE for the file till Cc is holding a reference to
it.
> >
> > On CLEANUP, try to do:
> > - CcFlushCache
> > then
> > - MmFlushImageSection
> > - CcPurgeCacheSection
> > - CcUninitializeCacheMap
> >
> > The cache map for the file will be placed to the “dead” list by the Cc,
Cc
> > will then delete the cache map and dereference the file object, thus
> causing
> > CLOSE to be sent.
> > Usually it will occur rather swiftly - provided you have called all of
the
> > above calls on CLEANUP.
> >
> > Max
> >
> >
> > —
> > You are currently subscribed to ntfsd as: xxxxx@Mikron.RU
> > To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com
> >
>
>
> —
> You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com
>

—
You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com