Disk file open is, most likely, synchronous operation anyway. You want to run multiple opens in parallel: run them on separate threads.
for a remote network-based file the networking part of it will surely be
asynchronous.
It can take a long time, so it becomes difficult to gauge how big a
thread pool to use to guarantee to not block clients due to slow opening
files for other clients.
I’ve seen some documentation which hints at MJ_CREATE possibly returning
STATUS_PENDING, I wonder if this is more likely with the redirector.
------ Original Message ------
From: “xxxxx@broadcom.com”
To: “Windows System Software Devs Interest List”
Sent: 19/05/2015 11:33:12 a.m.
Subject: RE:[ntdev] How to asynchronously create a file
>Disk file open is, most likely, synchronous operation anyway. You want
>to run multiple opens in parallel: run them on separate threads.
>
>—
>NTDEV is sponsored by OSR
>
>Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
>OSR is HIRING!! See http://www.osr.com/careers
>
>For our schedule of WDF, WDM, debugging and other seminars visit:
>http://www.osr.com/seminars
>
>To unsubscribe, visit the List Server section of OSR Online at
>http://www.osronline.com/page.cfm?name=ListServer
The CreateFile API does not accept an OVERLAPPED structure, which means that it is always going to be a synchronous operation. It is true that drivers may return STATUS_PENDING while processing an IRP_MJ_CREATE, but the I/O Manager will then simply wait for the I/O request to complete.
-scott
OSR
@OSRDrivers
what about for a kernel driver then? Surely it can do an IRP
asynchronously for this, and expose that to user mode via some other API
which can use completions?
Adrien
------ Original Message ------
From: “xxxxx@osr.com”
To: “Windows System Software Devs Interest List”
Sent: 19/05/2015 12:43:13 p.m.
Subject: RE:[ntdev] How to asynchronously create a file
>
>
>The CreateFile API does not accept an OVERLAPPED structure, which means
>that it is always going to be a synchronous operation. It is true that
>drivers may return STATUS_PENDING while processing an IRP_MJ_CREATE,
>but the I/O Manager will then simply wait for the I/O request to
>complete.
>
>-scott
>OSR
>@OSRDrivers
>
>—
>NTDEV is sponsored by OSR
>
>Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
>OSR is HIRING!! See http://www.osr.com/careers
>
>For our schedule of WDF, WDM, debugging and other seminars visit:
>http://www.osr.com/seminars
>
>To unsubscribe, visit the List Server section of OSR Online at
>http://www.osronline.com/page.cfm?name=ListServer
When pending is returned it also allows for the create to be canceled with CancelIoEx
d
Bent from my phone
From: xxxxx@osr.commailto:xxxxx
Sent: ?5/?18/?2015 5:43 PM
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: RE:[ntdev] How to asynchronously create a file
The CreateFile API does not accept an OVERLAPPED structure, which means that it is always going to be a synchronous operation. It is true that drivers may return STATUS_PENDING while processing an IRP_MJ_CREATE, but the I/O Manager will then simply wait for the I/O request to complete.
-scott
OSR
@OSRDrivers
—
NTDEV is sponsored by OSR
Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx></mailto:xxxxx>
> And then? so, after the page has its buffer heads attached to it, the IO for it
is NOT going thru the FSD at all?
As Martin pointed out already, this Linux-related discussion is just a “confusing noise” on this thread. Therefore, I will try to be quick.
Basically, all page-based IO is handled by FS-specific methods that are specified in address_space object. In practice, what these methods normally do is just deferring the whole thing to some generic buffer-head layer function (mpage_readpage(), mpage_writepages(),
__block_write_begin(),etc). All these functions take a pointer to get_block() callback that is implemented by the FS. The purpose of this callabck function is to translate every block in the page to a block device address, which does not need to be done more than once. In practice, this means that any subsequent involvement of FSD in paged IO is TRULY nominal - the whole thing gets handled by buffer head layer.
This is how it NORMALLY works. However, it does not necessarily HAVE to work this way. Let’s say you want to implement some advanced ZFS-like transactional filesystem. In such case it may well happen that the entire buffer-head layer functionality does not suit your objectives. Therefore, you may want to decide to implement all address_space object. methods in your very own way
This means that FSFs on Linux are not only very complex.
This means that FSFs on Linux are impossible.
Assuming that the term “FSF” refers to “file system filter”, what makes you believe that this Windows-specific feature may be needed by someone in “non-Windows world”, in the first place???
In any case, you are right - file system filters as they are understood in the Windows world are, indeed, impossible under Linux, simply because Linux VFS is just not designed this way,again, simply because such functionality is not needed under Linux
However, if you are just desperate to do things “the Windows way”…well, you DO remember “By programmers, for programmers” principle, right? in such case the only thing that you have to do is to write your very own “stacked/filtered filesystem API layer”. It may sound horrible - after all, you don’t want to upset the existing VFS code, do you? However, in actuality, you can do everything without modifying a SINGLE existing line of code. All you have to do is to present your API layer to VFS as just yet another filesystem, and that’ it. At this point you may start writing your own layered filesystems that DO want to be filtered, as well as filters, using your “new API”.
Sounds exciting, don’t you think…
Anton Bassov
>… the current design of file caches in Linux (a topic I profess total ignorance to)
Get educated then - after all, some day you may encounter “What is a Windows equivalent of generic_read_file()” question on NTDEV. What are you going to do then??? You remember the “kcalloc() equivalent” thread, don’t you…
Anton Bassov
>I’ve seen some documentation which hints at MJ_CREATE possibly returning
STATUS_PENDING, I wonder if this is more likely with the redirector.
Surely SMB file opens are cancellable in Vista+, I’m not sure on whether they can be async, but probably so.
–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com
------ Original Message ------
From: “xxxxx@broadcom.com”
To: “Windows System Software Devs Interest List”
Sent: 19/05/2015 11:33:12 a.m.
Subject: RE:[ntdev] How to asynchronously create a file
>Disk file open is, most likely, synchronous operation anyway. You want
>to run multiple opens in parallel: run them on separate threads.
>
>—
>NTDEV is sponsored by OSR
>
>Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
>OSR is HIRING!! See http://www.osr.com/careers
>
>For our schedule of WDF, WDM, debugging and other seminars visit:
>http://www.osr.com/seminars
>
>To unsubscribe, visit the List Server section of OSR Online at
>http://www.osronline.com/page.cfm?name=ListServer
>what about for a kernel driver then? Surely it can do an IRP
asynchronously for this
I don’t think you can roll your own CREATE IRP without using Io/Flt/ZwCreateFile.
File object setup and similar stuff in this path will be extremely hard to reproduce correctly.
–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com
More so, the thread pool can be implemented using IOCP.
“Marion Bond” wrote in message news:xxxxx@ntdev…
One of the basic properties of an IOCP is to allow additional threads to proceed when others are blocked. The thread pool APIs encapsulate this for you, so need not really worry about these details unless you really have nothing else to worry about
Sent from Surface Pro
From: Adrien de Croy
Sent: Monday, May 18, 2015 8:48 PM
To: Windows System Software Devs Interest List
what about for a kernel driver then? Surely it can do an IRP
asynchronously for this, and expose that to user mode via some other API
which can use completions?
Adrien
------ Original Message ------
From: “xxxxx@osr.com”
To: “Windows System Software Devs Interest List”
Sent: 19/05/2015 12:43:13 p.m.
Subject: RE:[ntdev] How to asynchronously create a file
>
>
>The CreateFile API does not accept an OVERLAPPED structure, which means
>that it is always going to be a synchronous operation. It is true that
>drivers may return STATUS_PENDING while processing an IRP_MJ_CREATE,
>but the I/O Manager will then simply wait for the I/O request to
>complete.
>
>-scott
>OSR
>@OSRDrivers
>
>—
>NTDEV is sponsored by OSR
>
>Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
>OSR is HIRING!! See http://www.osr.com/careers
>
>For our schedule of WDF, WDM, debugging and other seminars visit:
>http://www.osr.com/seminars
>
>To unsubscribe, visit the List Server section of OSR Online at
>http://www.osronline.com/page.cfm?name=ListServer
—
NTDEV is sponsored by OSR
Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer