Asynchronously create a file

hi all

apologies if this has already been asked on this list a million times, I
did quite a bit of searching.

We are working to refactor a lot of code to use IOCP for performance and
scalability with thread pools etc. All the documentation states not to
do anything blocking in a worker thread. However you hit a roadblock
when trying to open/create a lot of files over network paths.

Is there any way to asynchronously obtain a file handle? Spawning a
thread for this seems to fly in the face of proper thread pool
provisioning.

Is it really that big a problem or do I need to look into things like

  1. writing an entire SMB stack asynchronously so I can open remote files
    asynchronously.
  2. accessing zwCreateFile somehow in a way that allows me to be
    signalled when the file handle becomes available. E.g. write a driver.
    Would this even be possible? It looks like the IRP for file creation
    can be asynchronous.

Thanks

Adrien

Try using WT_EXECUTELONGFUNCTION flag in QueueUserWorkItem.

No, the I/O Manager always forces opens to be synchronous.

Drivers are free to return STATUS_PENDING for the create operation, but the
I/O Manager is going to wait for the i/O to complete before returning back
to the caller (happens in IopParseDevice). Even if you could somehow get
around this, Filter Manager will force the operation to be synchronous
anyway (it waits so that it can call the filter drivers in the same context
for both Pre- and Post-Create).

-scott
OSR
@OSRDrivers

“Adrien de Croy” wrote in message news:xxxxx@ntdev…
hi all

apologies if this has already been asked on this list a million times, I did
quite a bit of searching.

We are working to refactor a lot of code to use IOCP for performance and
scalability with thread pools etc. All the documentation states not to do
anything blocking in a worker thread. However you hit a roadblock when
trying to open/create a lot of files over network paths.

Is there any way to asynchronously obtain a file handle? Spawning a thread
for this seems to fly in the face of proper thread pool provisioning.

Is it really that big a problem or do I need to look into things like

1. writing an entire SMB stack asynchronously so I can open remote files
asynchronously.
2. accessing zwCreateFile somehow in a way that allows me to be signalled
when the file handle becomes available. E.g. write a driver. Would this
even be possible? It looks like the IRP for file creation can be
asynchronous.

Thanks

Adrien

THERE’s an abbreviation I haven’t seen before. Google tells me we’re talking about I/O Completion Ports, is that right?

So we’re talking user mode here, is that right?

If so (that’s fine… system programming extends to user mode certainly)… But I have to ask: why not just spin your own threads to attend to the completion ports, and not worry about what you should or should not do in the system-provided work threads? Assuming you’re not writing your code in basic C, then firing-off a few threads will be super-easy.

Peter
OSR
@OSRDrivers

ok thanks for that - so there’s actually a reason for it after all.

Maybe in Windows 11 they will add it.

Regards

Adrien

------ Original Message ------
From: “Scott Noone”
To: “Windows System Software Devs Interest List”
Sent: 4/11/2015 2:52:19 a.m.
Subject: Re:[ntdev] Asynchronously create a file

>


>
>No, the I/O Manager always forces opens to be synchronous.
>
>Drivers are free to return STATUS_PENDING for the create operation, but
>the I/O Manager is going to wait for the i/O to complete before
>returning back to the caller (happens in IopParseDevice). Even if you
>could somehow get around this, Filter Manager will force the operation
>to be synchronous anyway (it waits so that it can call the filter
>drivers in the same context for both Pre- and Post-Create).
>
>-scott
>OSR
>@OSRDrivers
>
>“Adrien de Croy” wrote in message
>news:xxxxx@ntdev…
>hi all
>
>apologies if this has already been asked on this list a million times,
>I did quite a bit of searching.
>
>We are working to refactor a lot of code to use IOCP for performance
>and scalability with thread pools etc. All the documentation states
>not to do anything blocking in a worker thread. However you hit a
>roadblock when trying to open/create a lot of files over network paths.
>
>Is there any way to asynchronously obtain a file handle? Spawning a
>thread for this seems to fly in the face of proper thread pool
>provisioning.
>
>Is it really that big a problem or do I need to look into things like
>
>1. writing an entire SMB stack asynchronously so I can open remote
>files asynchronously.
>2. accessing zwCreateFile somehow in a way that allows me to be
>signalled when the file handle becomes available. E.g. write a driver.
>Would this even be possible? It looks like the IRP for file creation
>can be asynchronous.
>
>Thanks
>
>Adrien
>
>—
>NTDEV is sponsored by OSR
>
>Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
>OSR is HIRING!! See http://www.osr.com/careers
>
>For our schedule of WDF, WDM, debugging and other seminars visit:
>http://www.osr.com/seminars
>
>To unsubscribe, visit the List Server section of OSR Online at
>http://www.osronline.com/page.cfm?name=ListServer

yes I/O Completion ports in Ring 3.

It basically looks like you need to hand it off to a thread. Presumably
the blocking is at least efficient and waiting threads are not
contributing to context switch hell, so maybe it’s not such a bad thing
to spawn threads for that.

I don’t think I’ll take the roll-my-own-SMB-stack option.

Cheers

Adrien

------ Original Message ------
From: “xxxxx@osr.com
To: “Windows System Software Devs Interest List”
Sent: 4/11/2015 8:21:10 a.m.
Subject: RE:[ntdev] Asynchronously create a file

>


>
>THERE’s an abbreviation I haven’t seen before. Google tells me we’re
>talking about I/O Completion Ports, is that right?
>
>So we’re talking user mode here, is that right?
>
>If so (that’s fine… system programming extends to user mode
>certainly)… But I have to ask: why not just spin your own threads to
>attend to the completion ports, and not worry about what you should or
>should not do in the system-provided work threads? Assuming you’re not
>writing your code in basic C, then firing-off a few threads will be
>super-easy.
>
>Peter
>OSR
>@OSRDrivers
>
>
>
>—
>NTDEV is sponsored by OSR
>
>Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
>OSR is HIRING!! See http://www.osr.com/careers
>
>For our schedule of WDF, WDM, debugging and other seminars visit:
>http://www.osr.com/seminars
>
>To unsubscribe, visit the List Server section of OSR Online at
>http://www.osronline.com/page.cfm?name=ListServer

CreateFile is a Synchronous call, but you will remember that reading the IOCP documentation in MSDN the OS is smart enough to release additional threads if the ones calling GetQueuedCompletionStatus block, but an even easier approach is to use the new (in the last 10 years) thread pool APIs that handle nearly anything you need to worry about directly

Sent from Mailhttp: for Windows 10

From: Adrien de Croy
Sent: November 2, 2015 5:40 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Asynchronously create a file

hi all

apologies if this has already been asked on this list a million times, I did quite a bit of searching.

We are working to refactor a lot of code to use IOCP for performance and scalability with thread pools etc. All the documentation states not to do anything blocking in a worker thread. However you hit a roadblock when trying to open/create a lot of files over network paths.

Is there any way to asynchronously obtain a file handle? Spawning a thread for this seems to fly in the face of proper thread pool provisioning.

Is it really that big a problem or do I need to look into things like

1. writing an entire SMB stack asynchronously so I can open remote files asynchronously.
2. accessing zwCreateFile somehow in a way that allows me to be signalled when the file handle becomes available. E.g. write a driver. Would this even be possible? It looks like the IRP for file creation can be asynchronous.

Thanks

Adrien


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</http:>