Copying 2 large files causes deadlock

Hello All -

We have an FSD that has a user-mode piece. The system gives the FSD a
write IRP, the fsd then writes the data to cache (CcCopyWrite). When CC
is flushing dirty pages, CC tells the FSD to write, then the FSD tells the
user-mode “server” to write the cached data to disk. This disk is a real
device, cached by the system, too. So each write results in duplicate
data in cache.

Only Big files demonstrate the problem. With small files (I don’t know
how small exactly, but I think it would be anything up to 4MB because that
is our dirty page limit (set via CcSetDirtyPageThreshold(
IrpSp->FileObject, 1000)) it doesn’t seem to be a problem. Writes to one
file work ok, but if two simultaneous writes to different files are going
on, within a minute or so the copying of data to both files stops. Windbg
shows that 2 FSD threads are waiting on seperate events, which the
user-mode server is supposed to set when it is done writing the block of
data.

Tracing the user-mode server shows that it gets to the WriteFile API call
and never returns from the OS, so it can’t ever set the event to allow the
FSD to process more data. This appears to be a classic deadlock
situation.

In the Cached write, I check for CcCanIWrite, and the code is basically
modelled after FASTFAT write example.

Also, the user-mode threads that are writing the data are running at a
THREAD_PRIORITY_TIME_CRITICAL priority. We’ve also tried
THREAD_PRIORITY_LOWEST priority, with similar results.

We thought maybe the system is running out of memory, but task manager
shows lots of available cache memory.

What could be causing this? How should we be doing these writes?

Thanks in advance - Greg

Are the threads you are blocking in your FSD the MPW thread? Sounds like
you are blocking system threads which are later needed to satisfy the
user mode write.

Pete

Peter Scott
xxxxx@KernelDrivers.com
http://www.KernelDrivers.com

>-----Original Message-----
>From: xxxxx@lists.osr.com [mailto:bounce-ntfsd-
>xxxxx@lists.osr.com] On Behalf Of Greg Pearce
>Sent: Wednesday, August 21, 2002 12:54 PM
>To: File Systems Developers
>Subject: [ntfsd] Copying 2 large files causes deadlock
>
>Hello All -
>
>We have an FSD that has a user-mode piece. The system gives the FSD a
>write IRP, the fsd then writes the data to cache (CcCopyWrite). When
CC
>is flushing dirty pages, CC tells the FSD to write, then the FSD tells
the
>user-mode “server” to write the cached data to disk. This disk is a
real
>device, cached by the system, too. So each write results in duplicate
>data in cache.
>
>Only Big files demonstrate the problem. With small files (I don’t
know
>how small exactly, but I think it would be anything up to 4MB because
that
>is our dirty page limit (set via CcSetDirtyPageThreshold(
>IrpSp->FileObject, 1000)) it doesn’t seem to be a problem. Writes to
one
>file work ok, but if two simultaneous writes to different files are
going
>on, within a minute or so the copying of data to both files stops.
Windbg
>shows that 2 FSD threads are waiting on seperate events, which the
>user-mode server is supposed to set when it is done writing the block
of
>data.
>
>Tracing the user-mode server shows that it gets to the WriteFile API
call
>and never returns from the OS, so it can’t ever set the event to allow
the
>FSD to process more data. This appears to be a classic deadlock
>situation.
>
>In the Cached write, I check for CcCanIWrite, and the code is
basically
>modelled after FASTFAT write example.
>
>Also, the user-mode threads that are writing the data are running at a
>THREAD_PRIORITY_TIME_CRITICAL priority. We’ve also tried
>THREAD_PRIORITY_LOWEST priority, with similar results.
>
>We thought maybe the system is running out of memory, but task manager
>shows lots of available cache memory.
>
>What could be causing this? How should we be doing these writes?
>
>Thanks in advance - Greg
>
>—
>You are currently subscribed to ntfsd as: xxxxx@KernelDrivers.com
>To unsubscribe send a blank email to %%email.unsub%%

Peter - That does appear to be what’s going on… I see that I’m holding
up the MPW threads. Now to figure out what to do about it… anyone have
a suggestion?

Thanks Peter and everyone else…

Greg

If you don’t need the IO request to the user service to be synchronous
to the MPW thread, since the MPW is already asynchronous to the original
write anyway, just queue it up to a worker thread and allow the MPW
threads to complete.

Pete

Peter Scott
xxxxx@KernelDrivers.com
http://www.KernelDrivers.com

>-----Original Message-----
>From: xxxxx@lists.osr.com [mailto:bounce-ntfsd-
>xxxxx@lists.osr.com] On Behalf Of Greg Pearce
>Sent: Friday, August 23, 2002 2:27 PM
>To: File Systems Developers
>Subject: [ntfsd] RE: Copying 2 large files causes deadlock
>
>Peter - That does appear to be what’s going on… I see that I’m
holding
>up the MPW threads. Now to figure out what to do about it… anyone
have
>a suggestion?
>
>Thanks Peter and everyone else…
>
>Greg
>
>—
>You are currently subscribed to ntfsd as: xxxxx@KernelDrivers.com
>To unsubscribe send a blank email to %%email.unsub%%

I thought of another way to solve this problem. That is by rolling my own
IRP, (which would be a copy of the one I got from the MPW Thread). I’d
queue the new IRP to one of my worker threads. Then I’d complete the MPW
IRP with status success and allow the MPW thread to continue doing its
thing.

Is this ok to do? How can I percolate an error back up to the system if
the actual write doesn’t work?

Is this where “Status_more_processing_required” might be useful?

Thanks again,

Greg

Yeah, if you want to re-use your Irp, then STATUS_M_P_R returned from
your completion routine will do the trick. As for returning an error
back to the system if your user service fails, see IoRaiseHardError().

Pete

Peter Scott
xxxxx@KernelDrivers.com
http://www.KernelDrivers.com

>-----Original Message-----
>From: xxxxx@lists.osr.com [mailto:bounce-ntfsd-
>xxxxx@lists.osr.com] On Behalf Of Greg Pearce
>Sent: Tuesday, August 27, 2002 9:41 AM
>To: File Systems Developers
>Subject: [ntfsd] RE: Copying 2 large files causes deadlock
>
>I thought of another way to solve this problem. That is by rolling my
own
>IRP, (which would be a copy of the one I got from the MPW Thread).
I’d
>queue the new IRP to one of my worker threads. Then I’d complete the
MPW
>IRP with status success and allow the MPW thread to continue doing its
>thing.
>
>Is this ok to do? How can I percolate an error back up to the system
if
>the actual write doesn’t work?
>
>Is this where “Status_more_processing_required” might be useful?
>
>Thanks again,
>
>Greg
>
>—
>You are currently subscribed to ntfsd as: xxxxx@KernelDrivers.com
>To unsubscribe send a blank email to %%email.unsub%%

Peter -

Thanks for the advice! I’m not sure I understand what you mean by
“re-use” “my” IRP. Can you clrify that?

Thanks - Greg

When you return STATUS_MORE_PROCESSING_REQUIRED from your completion
routine, the IO manager no longer is concerned with your Irp. Therefore,
if you were to, say, allocate a pool of Irps that you would use to send
these async requests to the user service, when this Irp was completed,
in your completion routine, you could return STATUS_M_P_R and then add
the Irp back to your pool. If you were to return some other status in
the completion routine then the Irp would be torn down as normal.

Hope this helps,

Pete

Peter Scott
xxxxx@KernelDrivers.com
http://www.KernelDrivers.com

>-----Original Message-----
>From: xxxxx@lists.osr.com [mailto:bounce-ntfsd-
>xxxxx@lists.osr.com] On Behalf Of Greg Pearce
>Sent: Tuesday, August 27, 2002 11:39 AM
>To: File Systems Developers
>Subject: [ntfsd] RE: Copying 2 large files causes deadlock
>
>Peter -
>
>Thanks for the advice! I’m not sure I understand what you mean by
>“re-use” “my” IRP. Can you clrify that?
>
>Thanks - Greg
>
>—
>You are currently subscribed to ntfsd as: xxxxx@KernelDrivers.com
>To unsubscribe send a blank email to %%email.unsub%%

Peter -

Yes! That helps! My driver is a FSD though, not a Filter Driver. I
don’t have a formal completion routine, per se. Is STATUS_M_P_R only for
filters?

Thanks again!

Greg

No, your FSD can have a separate pool of Irps it grabs from to send to
the user service. Eliminating the in-line allocation of the Irps, which
may not be bad in your application. It is in these Irps’ completion
routines that you would return STATUS_M_P_R in order to re-insert them
to your pool.

The actual status code is an indicator to the IO Manager to not continue
the processing of this particular Irp.

Pete

Peter Scott
xxxxx@KernelDrivers.com
http://www.KernelDrivers.com

>-----Original Message-----
>From: xxxxx@lists.osr.com [mailto:bounce-ntfsd-
>xxxxx@lists.osr.com] On Behalf Of Greg Pearce
>Sent: Tuesday, August 27, 2002 12:37 PM
>To: File Systems Developers
>Subject: [ntfsd] RE: Copying 2 large files causes deadlock
>
>Peter -
>
>Yes! That helps! My driver is a FSD though, not a Filter Driver. I
>don’t have a formal completion routine, per se. Is STATUS_M_P_R only
for
>filters?
>
>Thanks again!
>
>Greg
>
>—
>You are currently subscribed to ntfsd as: xxxxx@KernelDrivers.com
>To unsubscribe send a blank email to %%email.unsub%%

Thanks Pete - I’m trying this now, but I’ve got a buffer problem. I
reposted this today with a little more detail. Thanks for your help!

Greg

OK, yes you do have to re-allocate the buffer and copy over data. Since
completing the original Irp tells the system that you are done with the
request, and hence, the Mdl/buffer would potentially be torn down.

Pete

Peter Scott
xxxxx@KernelDrivers.com
http://www.KernelDrivers.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Greg Pearce
Sent: Wednesday, September 04, 2002 9:14 AM
To: File Systems Developers
Subject: [ntfsd] RE: Copying 2 large files causes deadlock

Thanks Pete - I’m trying this now, but I’ve got a buffer
problem. I reposted this today with a little more detail.
Thanks for your help!

Greg


You are currently subscribed to ntfsd as:
xxxxx@KernelDrivers.com To unsubscribe send a blank email to
%%email.unsub%%