IoTryQueueWorkItem appears in Win8 and is entirely undocumented. I’m
curious as to the conditions under which it will fail in its attempt to
queue a work item.
This function, for the curious, gets used in the equally undocumented
“forward progress” interface for UsbPort/UsbHub drivers in win8+, making my
life miserable.
Mark Roddy
I just refuse to believe my own eyes…
Since when are MVPs getting interested in anything undocumented, rather than simply doing what they are told by MSFT and strongly advising the rest of the world to follow the same path??? I just wonder what is going to happen next. If things keep on moving at such pace I would not be that much surprized if Mr.Burn disassembles the function in question and shares his experience with the rest of the world…
Anton Bassov
Anton,
You really don’t get it. First neither Mark or I are MVP’s anymore,
since Microsoft dropped the WDK MVP program. Second, just because we
advocate not using undocumented calls, does not mean we don’t research them.
In fact, this is one of those things like hooking, no you should not use
undocumented API’s but there are times that one needs to. I’ve delivered
several products to clients that use undocumented, or partially documented
(the include contains some or all of the interface but Microsoft does not
have a description), but when I do this I make sure that:
What I really get pissed off on and denounce are people who use the
undocumented API’s when they don’t have to.
Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@hotmail.com
Sent: Thursday, April 10, 2014 2:22 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] IoTryQueueWorkItem
I just refuse to believe my own eyes…
Since when are MVPs getting interested in anything undocumented, rather
than simply doing what they are told by MSFT and strongly advising the rest
of the world to follow the same path??? I just wonder what is going to
happen next. If things keep on moving at such pace I would not be that much
surprized if Mr.Burn disassembles the function in question and shares his
experience with the rest of the world…
Anton Bassov
NTDEV is sponsored by OSR
Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
Anton, did you even read what I wrote? I asked under what conditions
IoTryQueueWorkItem would fail. Did I say anywhere I am calling the
function? Oh wait, no I didn’t.
Thanks for playing.
Mark Roddy
On Thu, Apr 10, 2014 at 2:21 PM, wrote:
>
>
> I just refuse to believe my own eyes…
>
> Since when are MVPs getting interested in anything undocumented, rather
> than simply doing what they are told by MSFT and strongly advising the rest
> of the world to follow the same path??? I just wonder what is going to
> happen next. If things keep on moving at such pace I would not be that much
> surprized if Mr.Burn disassembles the function in question and shares his
> experience with the rest of the world…
>
>
>
>
> Anton Bassov
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> Did I say anywhere I am calling the function?
But I did not even suggest this…
Thanks for playing
OK, I have to admit that I meant to be a bit of an arsehole from the very beginning and decided to take the piss out of “good citizens”…
Concerning your actual question, I would rather put it a bit differently - I would ask “what is the purpose of such a function, in the first place”. The question is not as stupid as it seems to be. Let’s say your call has failed for some reason (for example, workitem queue got too large; all system threads that process workitems are currently blocked, etc). What are you supposed to do then if you are running at DPC level? The only possible solution in such situation seems to be to enqueue a low-priority DPC that subsequently tries to re-queue a workitem (and keeps on re - queueing itself in case of failure until it manages to queue a workitem).
It does not seem to be terribly efficient approach to allow failing to enqueue a workitem if the scenario has to be resolved by queueing a DPC instead, don’t you think…
Anton Bassov
> Allow priority DPC? Anton, WTF is wrong with you tonight?
I am just considering the situation when you call IoTryQueueWorkItem()from DPC routine and your call fails because, say, all threads that process workitems are currently blocked. How are you going to deal with this situation if you have to enqueue your workitem no matter what??? The only possible solution seems to be to queue a low-priority DPC (i.e. the one that gets executed when CPU has nothing else to do at the moment) and to try to enqueue your workitem again from this DPC routine…
As long as you use IoQueueWorkItem() or IoQueueWorkItemEx() you will never encounter this situation -
these functions are void and, hence, cannot fail. However, the name TryXXXX strongly suggests that
your call may fail. My point is that allowing a routine that is meant to queue a workitem fail does not seem to be reasonable at all, which makes me wonder about the purpose of IoTryQueueWorkItem(), in the first place…
Anton Bassov
IoTryQueueWorkItem () and the similar ExTryQueueWorkItem () were not intended to be undocumented.
The intended use of the API is to improve guarantee forward progress scenarios where one has to post a work item to a worker thread. Normally this is often handled by components keeping an “emergency use” thread off to the side to handle these requests, but the disadvantage of this is that one can’t parallelize requests that (might) need to be guaranteed forward progress, at least not without spinning up a bunch of worker threads that are dedicated use only (which has obvious downsides if a bunch of components do this).
IoTryQueueWorkItem lets such a component opportunistically attempt to enqueue such a work item to the executive work item queue; if the queue is not oversubscribed (i.e., there’s a thread immediately available to pick up the request) then the function returns TRUE and the work item is enqueued to an available worker thread. Otherwise, the function returns FALSE and the work item is not enqueued; it is then up to the caller to fall back to their private, “emergency use” worker thread (as it is possible that all threads in the system-supplied work item queue could be tied up with an item that needs inpaging etc. to occur first).
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Mark Roddy
Sent: Thursday, April 10, 2014 10:31 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] IoTryQueueWorkItem
IoTryQueueWorkItem appears in Win8 and is entirely undocumented. I’m curious as to the conditions under which it will fail in its attempt to queue a work item.
This function, for the curious, gets used in the equally undocumented “forward progress” interface for UsbPort/UsbHub drivers in win8+, making my life miserable.
Mark Roddy
— NTDEV is sponsored by OSR Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev OSR is HIRING!! See http://www.osr.com/careers For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
Thanks for the explanation.
Mark Roddy
On Fri, Apr 11, 2014 at 1:38 AM, Skywing wrote:
> IoTryQueueWorkItem () and the similar ExTryQueueWorkItem () were not
> intended to be undocumented.
>
>
>
> The intended use of the API is to improve guarantee forward progress
> scenarios where one has to post a work item to a worker thread. Normally
> this is often handled by components keeping an “emergency use” thread off
> to the side to handle these requests, but the disadvantage of this is that
> one can’t parallelize requests that (might) need to be guaranteed forward
> progress, at least not without spinning up a bunch of worker threads that
> are dedicated use only (which has obvious downsides if a bunch of
> components do this).
>
>
>
> IoTryQueueWorkItem lets such a component opportunistically attempt to
> enqueue such a work item to the executive work item queue; if the queue is
> not oversubscribed (i.e., there’s a thread immediately available to pick up
> the request) then the function returns TRUE and the work item is enqueued
> to an available worker thread. Otherwise, the function returns FALSE and
> the work item is not enqueued; it is then up to the caller to fall back to
> their private, “emergency use” worker thread (as it is possible that all
> threads in the system-supplied work item queue could be tied up with an
> item that needs inpaging etc. to occur first).
>
>
>
> - S (Msft)
>
>
>
> From: xxxxx@lists.osr.com [mailto:
> xxxxx@lists.osr.com] *On Behalf Of *Mark Roddy
> Sent: Thursday, April 10, 2014 10:31 AM
>
> To: Windows System Software Devs Interest List
> Subject: [ntdev] IoTryQueueWorkItem
>
>
>
> IoTryQueueWorkItem appears in Win8 and is entirely undocumented. I’m
> curious as to the conditions under which it will fail in its attempt to
> queue a work item.
>
>
>
> This function, for the curious, gets used in the equally undocumented
> “forward progress” interface for UsbPort/UsbHub drivers in win8+, making my
> life miserable.
>
>
> Mark Roddy
>
> — NTDEV is sponsored by OSR Visit the list at:
> http://www.osronline.com/showlists.cfm?list=ntdev OSR is HIRING!! See
> http://www.osr.com/careers For our schedule of WDF, WDM, debugging and
> other seminars visit: http://www.osr.com/seminars To unsubscribe, visit
> the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
Why do you calculate buffer length differently when you allocate it?
Paging Diane … 
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Skywing
Sent: Thursday, April 10, 2014 11:38 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] IoTryQueueWorkItem
IoTryQueueWorkItem () and the similar ExTryQueueWorkItem () were not intended to be undocumented.
I’m sure Ken filed a bug on this already, if needed.
Diane is currently on vacation… otherwise, knowing Diane, she would already have posted here!!
Peter
OSR
@OSRDrivers
Off the top of my head, you can use it to attempt to push work to the worker pool in order to get better parallelization and if that fails enqueue it for your backup dedicated worker thread, or process it in the current thread and starve the caller.
You handle the failure the way you handle any resource allocation failure - either fail the request or fall back to using backup resources (which may involve waiting for them to become available and may be slower)
-p
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@hotmail.com
Sent: Thursday, April 10, 2014 9:28 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] IoTryQueueWorkItem
Allow priority DPC? Anton, WTF is wrong with you tonight?
I am just considering the situation when you call IoTryQueueWorkItem()from DPC routine and your call fails because, say, all threads that process workitems are currently blocked. How are you going to deal with this situation if you have to enqueue your workitem no matter what??? The only possible solution seems to be to queue a low-priority DPC (i.e. the one that gets executed when CPU has nothing else to do at the moment) and to try to enqueue your workitem again from this DPC routine…
As long as you use IoQueueWorkItem() or IoQueueWorkItemEx() you will never encounter this situation - these functions are void and, hence, cannot fail. However, the name TryXXXX strongly suggests that your call may fail. My point is that allowing a routine that is meant to queue a workitem fail does not seem to be reasonable at all, which makes me wonder about the purpose of IoTryQueueWorkItem(), in the first place…
Anton Bassov
NTDEV is sponsored by OSR
Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>
>
> I just refuse to believe my own eyes…
>
> Since when are MVPs getting interested in anything undocumented, rather
> than simply doing what they are told by MSFT and strongly advising the
> rest of the world to follow the same path??? I just wonder what is going
> to happen next. If things keep on moving at such pace I would not be that
> much surprized if Mr.Burn disassembles the function in question and shares
> his experience with the rest of the world…
>
>
>
It sounded to me like this is being called by some Windows component that
he calls, and the question is under what conditions it fails, because
whatever component he is calling is giving undefined and undocumented
behavior. It’s one thing to want to use an undocumented feature, and
another to try to use an interface that has undefined (that is,
undocumented) behavior and consequently the upper level behavior on which
he must depend is not predictable. Therefore, in an attempt to understand
the failure mode, a desperate debugging session seems to have revealed
that this undocumented DDI is the cause of the undocumented behavior.
joe
Anton Bassov
NTDEV is sponsored by OSR
Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
What would failure scenario of IoTryQueueWorkitem be?
Plain IoAllocateWorkitem/IoQueueWorkitem(Ex) assumes that the workitem is fully initialized and doesn’t require any more resources. Thus, it’s unlikely Try version fails because of low resources.
Would it check that the IO_WORKITEM is already queued? Then Try moniker is not quite appropriate.
Does it check that there is an idle workitem thread, and fails if all threads are currently busy? In this case, the usability of such workitems will be extremely limited. Anyway, if you have a backup code path (a dedicated thread?) to handle the failures of IoTryQueue, why not just use it? A dedicated thread provides a side benefit of complete serialization, which means all your slow path stuff doesn’t require synchronization.
Microsoft just needs to provide an easy way to run your own workitem thread (wrap all that thread creation and rundown, WI queue management, etc).
> Does it check that there is an idle workitem thread, and fails if all threads are currently busy?
I think it was told in this topic - they yes. Really so.
–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com
>> Does it check that there is an idle workitem thread, and fails if all threads
are currently busy?
I think it was told in this topic - they yes. Really so.
So to sum it up:
If a driver has the workitems that are so important to progress, that you use a function to queue them that could fail, why use that facility at all?
Well that was really my question. If in fact the failure case is “no idle
worker threads” and the failure processing is “send to emergency thread
pool of my own creation”, I probably am going to implement my own worker
thread pool and skip the system pool entirely.
Mark Roddy
On Sun, Apr 13, 2014 at 12:07 PM, wrote:
> >> Does it check that there is an idle workitem thread, and fails if all
> threads
> are currently busy?
>
> >I think it was told in this topic - they yes. Really so.
>
> So to sum it up:
>
> If a driver has the workitems that are so important to progress, that you
> use a function to queue them that could fail, why use that facility at all?
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
I am indeed on vacation until 4/25. Free WiFi access for tourists is getting better and better here in Japan, but I’m not letting it distract me from what’s important, like the Sapporo beer museum.
Sent from my Windows Phone
From: xxxxx@osr.commailto:xxxxx
Sent: ?4/?12/?2014 0:06
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: RE:[ntdev] IoTryQueueWorkItem
I’m sure Ken filed a bug on this already, if needed.
Diane is currently on vacation… otherwise, knowing Diane, she would already have posted here!!
Peter
OSR
@OSRDrivers
—
NTDEV is sponsored by OSR
Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx></mailto:xxxxx>