Writing to a file from multiple threads

Does your filter work with just memory mapped files? If so, then I’m not sure where to suggest that you look.

Does your test do non-cached or cached I/O? Is there a pattern in the data corruption?

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com


From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of ganesh pashupathi
Sent: Monday, November 13, 2006 12:14 PM
To: ntfsd redirect
Subject: [ntfsd] Writing to a file from multiple threads

Hi,

? I have written test application that writes / reads the data in a single file c:\temp\test.txt from multiple threads and also multiple instances of the these test applications run. In one of the threads the file is written using the memory mapped api and in the other thread it is written using the usual WriteFile api. With my minifilter loaded and running this test application, the file does not get encrypted correctly.?

? I do not understand the reason why my filter fails. Can you guys point out the high points of failure as to where I should look for to solve this issue? This lead would be very helpful.? ?

~ganesh?

— Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17 You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’ To unsubscribe send a blank email to xxxxx@lists.osr.com

I don;t know how it work on Minifilter world. But for legacy filter, check
the NOCACHE IRPs and do the encryption in write and decryption in read
dispatch routine. While writing make sure encrypting on a temporary buffer,
don;t change the original buffer contents.

Sisimon

On 14 Nov 2006 14:23:06 -0000, ganesh pashupathi <
xxxxx@rediffmail.com> wrote:

Hi Tony,

Many thanks for your reply.

My filter works for memory mapped test application (Visual Studio 2003
style. i.e. CreateFile -> CreateMapping -> Use the mapping to write to the
file -> Flush -> Unmapview -> mapping close -> file object set length)
individually (not multithreaded not multiapplication but I can write to the
several thousand times with varying lengths). It also works for the wordpad
style test application i.e. (CreateFile + WriteFile + CloseHandle)
individually(not multithreaded not multiapplication but I can write to the
several thousand times with varying lengths).

However, with the mix of the applications or multithreading or multiple
instances my filter fails for the individual applications as well as the
mix.

I have maintained some indicators in the per stream context for my
filter. These indicators seem to have different values then what I expect.
For e.g. I have a indicator which is set in the cached write which
indicates that the file size has been increased and now in the non cached
write I have to insert the header and shift the data that was present by the
size of header. This indicator is also reset in some other callbacks e.g.
precreate if FILE_WRITE_DATA. Is it possible that before the non cache write
comes in my flag is reset?

Kindly provide inputs on the same. This would be very helpful.

Thanks again,
~ganesh

On Tue, 14 Nov 2006 Tony Mason wrote :
>Does your filter work with just memory mapped files? If so, then I’m not
sure where to suggest that you look.
>
>Does your test do non-cached or cached I/O? Is there a pattern in the
data corruption?
>
>Tony
>
>Tony Mason
>Consulting Partner
>OSR Open Systems Resources, Inc.
>http://www.osr.com
>
>
>
>________________________________________
> From: xxxxx@lists.osr.com [mailto:
xxxxx@lists.osr.com] On Behalf Of ganesh pashupathi
>Sent: Monday, November 13, 2006 12:14 PM
>To: ntfsd redirect
>Subject: [ntfsd] Writing to a file from multiple threads
>
>Hi,
>
> I have written test application that writes / reads the data in a
single file c:\temp\test.txt from multiple threads and also multiple
instances of the these test applications run. In one of the threads the file
is written using the memory mapped api and in the other thread it is written
using the usual WriteFile api. With my minifilter loaded and running this
test application, the file does not get encrypted correctly.
>
> I do not understand the reason why my filter fails. Can you guys point
out the high points of failure as to where I should look for to solve this
issue? This lead would be very helpful.
>
>~ganesh
>
>
>— Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17 You are currently subscribed
to ntfsd as: unknown lmsubst tag argument: ‘’ To unsubscribe send a blank
email to xxxxx@lists.osr.com
>
>—
>Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
>
>You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’
>To unsubscribe send a blank email to xxxxx@lists.osr.com

http:— Questions? First check the IFS FAQ at
> https://www.osronline.com/article.cfm?id=17 You are currently subscribed
> to ntfsd as: unknown lmsubst tag argument: ‘’ To unsubscribe send a blank
> email to xxxxx@lists.osr.com


GCS d+ s: a- c++++ U> B+ L++>$ w++++$ W++(+++) PGP+N+ t PS+PE++ tv+(++) b+++
G+++ e++>(++++) h-- r
Don’t know this? See http://www.geekcode.com/geek.html</http:>

And I’m going to ask the most important of my questions again, because
it’s the one you didn’t answer:

In your test program, WHAT KIND OF I/O DO YOU USE?

The problem with the scenario you describe (consistency issues in
multi-threaded test environments) is that there are subtle assumptions
resting in here. For example, if you have two I/O operations:

Thread A: Write (0,4096)
Thread B: Write (2048, 6134)

The behavior of those I/O operations will depend very much on a variety
of things. If those are asynchronous you have two equally valid
outcomes, and hence two valid states of the file (remember, as the FSD
we do not need to inject deterministic behavior into a non-deterministic
system.)

If you mix memory mapped and non-cached I/O in multi-threaded
applications you would similarly see inconsistent results - this is
documented by Microsoft (although I heard rumblings that this might
change in the future, so non-cached and memory mapped I/O become
coherent.)

So this gives me my second question:

HOW DO YOU DEFINE FAILURE (OR SUCCESS)?

(Sorry for the shouting, but these questions weren’t answered previously
and certainly the first one was asked, if not the second one.)

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

Your test is flawed because it expects determinism in a system you’ve
constructed that is deliberately non-deterministic.

Conceptually, write operations are nothing more than RtlCopyMemory calls
from the user buffer to the cache buffer. Between *any* two bytes the
thread could lose the CPU and the other thread could gain it. Due to
the way that modern machines work, you’re likely to see cache line
granularities, but that’s a side-effect of the implementation.

When you run single threaded, the test is deterministic (only one change
at a time) but when you add multiple threads the test is
non-deterministic because you do not know the order in which the writes
are applied.

Try running your test program without your filter involved. Look at the
resulting file. If my theory is correct, you will notice that the file
contents are almost always jumbled.

You can “fix” this by adding determinism to your filter. I use the
quotation marks because this is a bad idea - applications that really do
asynchronous I/O are performance critical and will suffer serious
performance degradation if you serialize their I/O.

A better fix is to realize that what you are doing is itself
fundamentally flawed. Have the threads write to non-overlapping regions
of the file and see if that works for you. Or have them use byte range
locking to serialize with each other and confirm that resolves your
problem (no filter changes, just application changes.0

We are NOT responsible at file systems level for fixing broken
applications. They were broken before, and they can be broken after.

Hmm. The file systems mantra: “we don’t care about the user’s data”
although I usually say “so long as WE don’t corrupt it.”

Tony

Tony Mason

Consulting Partner

OSR Open Systems Resources, Inc.

http://www.osr.com http:</http:>


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of ganesh pashupathi
Sent: Wednesday, November 15, 2006 9:46 AM
To: ntfsd redirect
Subject: Re: RE: [ntfsd] Writing to a file from multiple threads

Hi Tony,

I am very sorry for not answering that question. Please except my
apologies.

I use cached I/O. Also in the filter I do not change all the requests
to non cache by setting FILE_FLAG_NO_BUFFERING (also
FILE_FLAG_WRITE_THROUGH is not set by me). The same is true for both the
test applications.

One of the application writes all ‘1’ in the file and the second
writes ‘2’ in the file. So success for me would be either all 1’s in the
file or 2’s in the file in encrypted format with header inserted.

Sorry and Thanks,
~ganesh

On Wed, 15 Nov 2006 Tony Mason wrote :

And I’m going to ask the most important of my questions again, because
it’s the one you didn’t answer:

In your test program, WHAT KIND OF I/O DO YOU USE?

The problem with the scenario you describe (consistency issues in
multi-threaded test environments) is that there are subtle assumptions
resting in here. For example, if you have two I/O operations:

Thread A: Write (0,4096)
Thread B: Write (2048, 6134)

The behavior of those I/O operations will depend very much on a variety
of things. If those are asynchronous you have two equally valid
outcomes, and hence two valid states of the file (remember, as the FSD
we do not need to inject deterministic behavior into a
non-deterministic
system.)

If you mix memory mapped and non-cached I/O in multi-threaded
applications you would similarly see inconsistent results - this is
documented by Microsoft (although I heard rumblings that this might
change in the future, so non-cached and memory mapped I/O become
coherent.)

So this gives me my second question:

HOW DO YOU DEFINE FAILURE (OR SUCCESS)?

(Sorry for the shouting, but these questions weren’t answered
previously
and certainly the first one was asked, if not the second one.)

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

http:ignature-home.htm/xxxxx@Middle5?PARTNER=3> — Questions? First
check the IFS FAQ at https://www.osronline.com/article.cfm?id=17 You are
currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’ To
unsubscribe send a blank email to xxxxx@lists.osr.com</http:>