CcCopyRead and CcCopyWrite don't finish if Length in bytes of the data is 0x80000 (512 KB) on Window

OSR_Community_User

Hi,

I have an issue to use CcCopyRead and CcCopyWrite, they don't finish if Length in bytes of the data is 0x80000 (512 KB) on Windows Vista, is there some limitation here?

Any suggestion are appreciate.

Thanks,

Tao
_________________________________________________________________
Try Chicktionary, a game that tests how many words you can form from the letters given. Find this and more puzzles at Live Search Games!
http://g.msn.ca/ca55/207

OSR_Community_User

What does it mean when you say that they "don't finish". Are you saying
that they hang? Or are you saying that they do a data transfer that is
different than requested? If it is the former, what do you see when you
look at the stack in the debugger? If it is the latter, how much data
do they transfer and what status do they return?



Is this to your own file system? If so, my suggestion is that you have
a bug in your file system. If not, I'd want to understand how you are
using CcCopyRead/CcCopyWrite (since they really should only be used by
the owner of the cache.)



Tony

OSR

Gabriel_Bercea

Did you acquire the FCB shared for reads and exclusively for writes ?

You must own the FCB before calling these functions.



From: [email protected]
[mailto:[email protected]] On Behalf Of Tony Mason
Sent: Wednesday, August 13, 2008 5:36 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] CcCopyRead and CcCopyWrite don't finish if Length in
bytes of the data is 0x80000 (512 KB) on Windows Vista



What does it mean when you say that they "don't finish". Are you saying
that they hang? Or are you saying that they do a data transfer that is
different than requested? If it is the former, what do you see when you
look at the stack in the debugger? If it is the latter, how much data do
they transfer and what status do they return?



Is this to your own file system? If so, my suggestion is that you have a
bug in your file system. If not, I'd want to understand how you are using
CcCopyRead/CcCopyWrite (since they really should only be used by the owner
of the cache.)



Tony

OSR




---
NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ''
To unsubscribe send a blank email to [email protected]

OSR_Community_User

> Did you acquire the FCB shared for reads and exclusively for writes ?

The latter is wrong.

Exclusive acquisition of FCB is only done for a write which increases the file
size.

Exclusive acquisiton of paging IO resource is only done in truncation path.

Anyway these fields are not intended to be accessed from filters. The FSD owns
them.

--
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
[email protected]
http://www.storagecraft.com

OSR_Community_User

Max's final point is the one people need to keep in mind:

"these fields are not intended to be accessed from filters. The FSD owns
them."

If you assume you know what the underlying file system's locking model
is, you're broken. If possible, at least restrict your broken software
to working with a single file system - that way the fact you are broken
is less likely to leak out.

Note: there is NO requirement that an FSD use either of these resources.
There are flags that indicate (for FsRtl's benefit) which resources ARE
used, but the locking callbacks have been added to give the file system
greater flexibility in building locking models. Outside FsRtl (which
explicitly requires a format to use its functions) ANY component that
reaches into FileObject->FsContext and assumes it knows the format of
those data structures is fundamentally broken.

I've spent a HUGE amount of time in the past few years dealing with
people that cavalierly violate the layering/isolation rules like this
and my tolerance for this mis-behavior is very low.

Tony
OSR

OSR_Community_User

Hi Tony,

I am working on Network Redirector Driver, the problem is hung, the strange thing is if Length in bytes of the data from CcCopyRead and CcCopyWrite is 0x10000 (64KB) or less, I don't have such problem, only for Length is 0x80000 (512 KB), change READ_AHEAD_GRANULARITY from 0x10000 (64KB) to 0x80000 (512 KB) inCcSetReadAheadGranularity, it doen't help.

Thanks,

Tao



> Subject: RE: [ntfsd] CcCopyRead and CcCopyWrite don't finish if Length in bytes of the data is 0x80000 (512 KB) on Windows Vista> Date: Wed, 13 Aug 2008 12:35:10 -0400> From: [email protected]> To: [email protected]> > Max's final point is the one people need to keep in mind:> > "these fields are not intended to be accessed from filters. The FSD owns> them."> > If you assume you know what the underlying file system's locking model> is, you're broken. If possible, at least restrict your broken software> to working with a single file system - that way the fact you are broken> is less likely to leak out.> > Note: there is NO requirement that an FSD use either of these resources.> There are flags that indicate (for FsRtl's benefit) which resources ARE> used, but the locking callbacks have been added to give the file system> greater flexibility in building locking models. Outside FsRtl (which> explicitly requires a format to use its functions) ANY component that> reaches into FileObject->FsContext and assumes it knows the format of> those data structures is fundamentally broken.> > I've spent a HUGE amount of time in the past few years dealing with> people that cavalierly violate the layering/isolation rules like this> and my tolerance for this mis-behavior is very low.> > Tony> OSR> > > ---> NTFSD is sponsored by OSR> > For our schedule debugging and file system seminars> (including our new fs mini-filter seminar) visit: > http://www.osr.com/seminars> > You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ''> To unsubscribe send a blank email to [email protected]
_________________________________________________________________

OSR_Community_User

Well, if it's hung, it means the thread are waiting for something to
happen. The question is "For what are they waiting?"



64KB is a magic number in virtual memory - it's the unit of granular
mapping in the memory manager. 256KB is a magic number as well as it's
the unit of address management in the cache manager. So when you see
something that "works" in one case but not in another and the units
involved are one of these it suggests you're invoking some additional
processing in one of these components (e.g., faulting in more data,
extending sections, creating new VACBs, etc.)



I've used CcCopyRead with values much larger than 64KB. It is not an
issue of the cache manager, but rather some operation your call is
invoking that is in turn relying on you (or someone else) to fulfill.
What does the debugger say? When you look at the stack trace of the
thread that called CcCopyRead, what does it say the thread is waiting to
have happen? If you figure that out, then try to determine why that
event is not occurring. It could be (for example) that you are getting
a callback into your file system driver to acquire a lock and you've
decided to use a non-reentrant lock (e.g., synchronization event or fast
mutex) or you've acquired an ERESOURCE in "shared" mode and then in the
same thread (via a callback) tried to acquire it in "exclusive" mode
(this *always* deadlocks and *never* tells you before it does so.)



Of course I assume you're running on the checked version of the NT
kernel and it's not reporting anything interesting. Naturally, you'd
also be using driver verifier and similarly *it* isn't reporting
anything. Finally, of course, since you didn't report anything unusual,
I assume that "!deadlock" said there was nothing it could find.



Tony

OSR

OSR_Community_User

Thank you for Tony's reply, we fix the issue.

The bug that we had before, but only show up on Windows Vista with SP1, look like without SP1, we are luck that still work, I think CopyFileEx is change in SP1, we got CcCopyRead and CcCopyWrite with large value as 512 KB, before we check if IrpContext->Wait is not wait, we will FsdPostRequest, this is not correct, it will wait for WrPageOut, but works with small value.

Thanks,

Tao




Subject: RE: [ntfsd] CcCopyRead and CcCopyWrite don't finish if Length in bytes of the data is 0x80000 (512 KB) on Windows VistaDate: Thu, 14 Aug 2008 13:53:46 -0400From: [email protected]: [email protected]




Well, if it?s hung, it means the thread are waiting for something to happen. The question is ?For what are they waiting??

64KB is a magic number in virtual memory ? it?s the unit of granular mapping in the memory manager. 256KB is a magic number as well as it?s the unit of address management in the cache manager. So when you see something that ?works? in one case but not in another and the units involved are one of these it suggests you?re invoking some additional processing in one of these components (e.g., faulting in more data, extending sections, creating new VACBs, etc.)

I?ve used CcCopyRead with values much larger than 64KB. It is not an issue of the cache manager, but rather some operation your call is invoking that is in turn relying on you (or someone else) to fulfill. What does the debugger say? When you look at the stack trace of the thread that called CcCopyRead, what does it say the thread is waiting to have happen? If you figure that out, then try to determine why that event is not occurring. It could be (for example) that you are getting a callback into your file system driver to acquire a lock and you?ve decided to use a non-reentrant lock (e.g., synchronization event or fast mutex) or you?ve acquired an ERESOURCE in ?shared? mode and then in the same thread (via a callback) tried to acquire it in ?exclusive? mode (this *always* deadlocks and *never* tells you before it does so.)

Of course I assume you?re running on the checked version of the NT kernel and it?s not reporting anything interesting. Naturally, you?d also be using driver verifier and similarly *it* isn?t reporting anything. Finally, of course, since you didn?t report anything unusual, I assume that ?!deadlock? said there was nothing it could find.

Tony
OSR
--- NTFSD is sponsored by OSR For our schedule debugging and file system seminars (including our new fs mini-filter seminar) visit: http://www.osr.com/seminars You are currently subscribed to ntfsd as: unknown lmsubst tag argument: '' To unsubscribe send a blank email to [email protected]
_________________________________________________________________

Vayne3000

A developer is having this same issue in a UDF File System Driver. How does he go about fixing it? The work he has done on it can be found at https://github.com/assorted/reactos/tree/udf.

Scott_Noone_(OSR)

If you have a specific problem open a new thread.