Hi Bruno,
I have a theory of what might be occurring here, but it will certainly
take some time to confirm that this is the case. Your additional
information about RXCE_SEND_SYNCHRONOUS certainly helps here - it
suggests that the post is occurring, not at file systems level, but
rather at transport/communications level.
Thus, my theory is that there are some low level operations that are
being posted by the transport. Have you looked at the SMB sample code
in the IFS Kit? It demonstrates some of this point (see
src\filesys\smbmrx\wnet\sys\sndrcv.c for example). The function
SmbCeSend appears to be called in some circumstances with the
RXCE_SEND_SYNCHRONOUS option, but not always. SmbCeTranceive appears to
always be called with this option set.
I suspect that RDBSS is never posting the request, but it does propagate
the return value if the communications layer posts it. It is quite
possible that post operations might be occurring as a result of other
side events, such as authentication, establishing a connection, etc. -
and not directly the open of the file. To determine this, it would
really require a simultaneous packet trace to examine as well.
Another possible testing mechanism is to breakpoint on the Smb
transmission operations and see if stripping that option bit will cause
it to post the operations (fair warning: the code above may not handle
posting the code properly, but if you do this in the debugger the worst
that will happen is it crashes. Perfectly reasonable for learning more
about how the system works in this area.)
I certainly understand your concern about launching thousands of threads
simultaneously to implement this functionality - but my suspicion here
is that the usage you are making was never envisioned by the redirector
team. That doesn’t make it an invalid usage, but does suggest that this
will prove to be rather challenging.
Since you indicate your customer is using the OSR toolkit, you should
also encourage them to use their support channel with us.
Regards,
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Bruno Sartirana
Sent: Monday, January 31, 2005 4:39 AM
To: ntfsd redirect
Subject: RE: [ntfsd] Can’t get the Windows Server 2003 redirector return
STATUS_PENDING for IRP_MJ_CREATE IRPs
Correction of " I can’t find a good reason for implementing such
operation synchronous when it involves accessing the network,
because it’s got to become asynchronous before TDI is called anyhow."
Actually the open/create thread can be suspended either in the
redirector or below the TDI.
Low-level MRXSMB output routines return STATUS_PENDING if the flag
RXCE_SEND_SYNCHRONOUS is not set in SendOptions.
It would be interesting to know what RDBSS does on an IRP_MJ_CREATE to
trigger the use of RXCE_SEND_SYNCHRONOUS in MRXSMB output
routines, and if its behavior can be modified by setting IRP
parameters/flags differently. There should be a way to extract a
STATUS_PENDING from RDBSS on a create, as I saw it returned in the
debugger (but not without it).
Bruno Sartirana
Apogeo, Inc.
http://www.apogeo.com
-----Original Message-----
From: Bruno Sartirana [mailto:xxxxx@apogeo.com]
Sent: Sunday, January 30, 2005 21:31
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Can’t get the Windows Server 2003 redirector return
STATUS_PENDING for IRP_MJ_CREATE IRPs
Hi Tony,
Even Microsoft is scrambling to find an answer to my question, and,
after a long and still inconclusive exchange with them, I
decided to post it here, counting on your stepping in.
I admit that, for convenience, I pre-set IRP_NOCACHE when I allocate my
IRPs for any type of requests. I have no other specific
reason to set it on IRP_MJ_CREATE (and the other two types of requests,
as you pointed out), but I thought it won’t hurt. I
mentioned it just to provide accurate information.
My customer uses your FSDK to implement a distributed FSD. When an FSDK
thread calls the FSD to create a file, and the file must be
created fragmented over a number of remote volumes, we found it
convenient to use that single thread to post, through the library I
wrote, all the necessary create IRPs to the redirector, in short
sequence, counting on the redirector returning STATUS_PENDING for
each create IRP.
The alternative would be to use worker threads, but I’m afraid that we
might have some difficulties in relying on this solution when
thousands of open/create requests are generated at the same time. For
example, if 5000 files are being opened that are fragmented
across 8 volumes, we would need 40000 threads to carry out the
operations. This number would double in case of single mirroring and
open-for-write operations. It could get worse for multiple mirroring.
I superficially checked the FAT file system sample in the IFS2003 Kit.
It clearly returns STATUS_PENDING under some circumstances.
Since the redirector can have very long latency on a file create that
requires going over the network, it seemed logical to expect
it would return STATUS_PENDING. I can’t find a good reason for
implementing such operation synchronous when it involves accessing
the network, because it’s got to become asynchronous before TDI is
called anyhow.
I have also to say that, if I set a debugger breakpoint in
rdbss!RxCommonCreate and manually examine the create IRPs I send down,
or
those that the I/O Manager sends down, the first I catch receives
STATUS_PENDING. I don’t know if the IoCompletion routine called by
the redirector for those IRPs reports STATUS_SUCCESS or some error due
to my messing up with the redirector’s timing. I’m getting
close to write a filter driver that attaches to the redirector to prove
or disprove once and for all, without possible timing
distortions created by the debugger, whether the redirector ever returns
STATUS_PENDING. I’m that desperate to know.
Let me also point out that I don’t understand why the redirector
requires the flag IPR_SYNCHRONOUS_API or it would assert, and why
the IFS2003 Kit documentation recommends (or requires?) to set both
IRP_DEFER_IO_COMPLETION and IRP_SYNCHRONOUS_API for
IRP_MJ_CREATE IRPs. Those are conflicting flags that, used together,
don’t add any information to the IRP. Either I’d like the
operation to be synchronous or asynchronous, knowing that it could be
carried out either way independently of my desire.
This a major stumbling block for us.
Thanks,
Bruno Sartirana
Apogeo, Inc.
http://www.apogeo.com
-----Original Message-----
From: Tony Mason [mailto:xxxxx@osr.com]
Sent: Sunday, January 30, 2005 20:09
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Can’t get the Windows Server 2003 redirector return
STATUS_PENDING for IRP_MJ_CREATE IRPs
Hi Bruno,
It is clear you’ve spent quite a bit of time studying this issue. One
thing to keep in mind here is that even if you unravel how this is
implemented on a particular version, there is no guarantee that the
behavior will not change. The “rules” for I/O operations allow a driver
to implement ANY call synchronously and (correspondingly) ANY call
asynchronously.
I can make several notes:
- IRP_NOCACHE should have no effect on IRP_MJ_CREATE,
IRP_MJ_QUERY_INFORMATION or IRP_MJ_SET_INFORMATION with respect to the
synchronous (or asynchronous) nature of the request.
- Most file systems use IoIsOperationSynchronous to determine if the
operation should be implemented synchronously, although this is often
complicated by other internal file system considerations. You might
wish to at least use this in your own testing of the interface(s).
- A driver is free to implement ALL I/O operations synchronously if it
so desires.
- A driver is free to implement ALL I/O operations asynchronously if it
so desires. (Note that the I/O Manager makes this case work properly.
Any driver that steps into the role of I/O Manager must preserve this
behavior.)
- IRP_MJ_CREATE is normally implemented synchronously. Indeed, I’m
surprised you are seeing ANY asynchronous create operations. That’s
certainly the suggestion of the ASSERT you are seeing in RDBSS.
Is there a reason that you must do this all within a single thread
context?
Regards,
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Bruno Sartirana
Sent: Thursday, January 27, 2005 11:36 PM
To: ntfsd redirect
Subject: [ntfsd] Can’t get the Windows Server 2003 redirector return
STATUS_PENDING for IRP_MJ_CREATE IRPs
I wrote a kernel library that is part of a distributed FSD based on the
FSDK. The FSD has to run on Server 2003 only.
The library exports functions like “xxxCreateFile” ,
“xxxSetFileInformation”, and so on. Those functions create the
appropriate
IRPs and send them directly to the redirector, as the target files are
only
on remote shares.
The library creates all IRPs with the IRP_NOCACHE flag, because we don’t
want any data caching by the redirector.
Everything works fine, except that IRP_MJ_CREATE,
IRP_MJ_SET_INFORMATION,
and IRP_MJ_QUERY_INFORMATION requests are always handled synchronously
by
the redirector. IRP_MJ_READ and IRP_MJ_WRITE requests always return
STATUS_PENDING, as expected.
Because a single thread needs to fire off multiple IRPs per file when
the
file is scattered across several remote volumes, we must find a way to
induce the redirector to return STATUS_PENDING.
Focusing on IRP_MJ_CREATE IRPs to start, according to the IFS2003, the
flags
to be specified are three:
. IRP_CREATE_OPERATION
. IRP_DEFER_IO_COMPLETION
. IRP_SYNCHRONOUS_API
As I said before, I also use IRP_NOCACHE. The IRP_MJ_CREATE requests
carrying those four flags, are always processed synchronously by the
redirector, even in cases when the files don’t exist and need to be
created.
By dropping IRP_SYNCHRONOUS_API, I get the following assertion from
RDBSS:
*** Assertion failed: Status != STATUS_PENDING
*** Source File: d:\srv03rtm\base\fs\rdr2\rdbss\create.c, line 3685
Break repeatedly, break Once, Ignore, terminate Process, or terminate
Thread
(boipt)?
and the IRPs get completed synchronously anyhow. So do they if I drop
both
IRP_DEFER_IO_COMPLETION and IRP_SYNCHRONOUS_API, although in this case,
if I
remember well, no assert is hit.
File create requests that don’t go through our FSD and library,
sometimes
are processed asynchronously by the redirector. For example, by setting
a
breakpoint in rdbss!RxCommonCreate, I could see that a console command
like
“type \server\share\file” reaches RxCommonCreate with an IRP_MJ_CREATE
IRP
with Flags=0x884 (IRP_CREATE_OPERATION | IRP_DEFER_IO_COMPLETION |
IRP_SYNCHRONOUS_API), which are the same I use, except for my other flag
IRP_NOACHE. RxCommonCreate sometimes replies with STATUS_PENDING to I/O
Manager-generated IRPs like that in the “type” command case, but not to
those I generate. This test proves, at least, that the redirector can
return
STATUS_PENDING on IRP_MJ_CREATE requests.
I collected statistics in my library for hundreds of file create
operations
for non-existent files: none was processed asynchronously by the
redirector.
As far as IRP_MJ_SET_INFORMATION and IRP_MJ_QUERY_INFORMATION are
concerned,
they are limited to setting and retrieving the file size. They use these
IRP
flags:
. IRP_NOCACHE
. IRP_BUFFERED_IO (the AssociatedIrp.SystemBuffer field is used)
They are always handled synchronously by the redirector.
In conclusion, what am I supposed to set in my IRP_MJ_CREATE,
IRP_SET_INFORMATION_FILE, and IRP_QUERY_INFORMATION_FILE IRPs to get
STATUS_PENDING from the redirector when it has to go over the network to
process the request?
Thanks,
Bruno Sartirana
Apogeo, Inc.
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@apogeo.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com