Completion Port Bypass

NTDEV
1

In the MSDN page on GetQueuedCompletionStatus, it describes a mechanism
for submitting a request on a completion port file handle that is
supposed to bypass the completion port, by setting the low-order bit of
the hEvent handle in the OVERLAPPED structure.

I’m seeing evidence that this mechanism may be broken in Windows 7,
although I find that hard to believe. My bypass-tagged requests are
getting returned on my completion port. The people on this list have
experience with high-performance I/O operations. Has anyone ever seen
such a thing?

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

2

Run the app under a debugger and see if you’re getting “invalid handle” exceptions.

The code that sets the bit, how exactly does it look?

3

xxxxx@broadcom.com wrote:

Run the app under a debugger and see if you’re getting “invalid handle” exceptions.

The code that sets the bit, how exactly does it look?

The requests themselves are going through just fine. I still get
results with GetOverlappedResult, the change is that I am ALSO getting a
notification on my completion port.

I’ve used this code in other projects on XP.

if( m_hPort )
ol.hEvent = (HANDLE)((ULONG_PTR)ol.hEvent | 1);

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

4

Does the problem happen in 32 bit Windows? In 64 bit Windows for 32 bit app?

With what device are you running the IO?

5

Tim,

I used it successfully on Server 2008, but I haven’t tried since
then.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tim Roberts
Sent: Monday, May 12, 2014 5:19 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Completion Port Bypass

xxxxx@broadcom.com wrote:

Run the app under a debugger and see if you’re getting “invalid handle”
exceptions.

The code that sets the bit, how exactly does it look?

The requests themselves are going through just fine. I still get results
with GetOverlappedResult, the change is that I am ALSO getting a
notification on my completion port.

I’ve used this code in other projects on XP.

if( m_hPort )
ol.hEvent = (HANDLE)((ULONG_PTR)ol.hEvent | 1);

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

6

xxxxx@broadcom.com wrote:

Does the problem happen in 32 bit Windows? In 64 bit Windows for 32 bit app?

Fair question; I know it’s 64-bit Windows, and I’m 99% sure it’s a
64-bit app, although I’m not clear why that would be relevant.

With what device are you running the IO?

It’s my driver for a custom PCIe device, but that should all be handled
at a layer above me.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

7

In the case of undesired completion notification, does it get posted when the I/O completes synchronously or with ERROR_IO_PENDING? Does it happen for every type of request, or for a specific IOCTL code only?

8

Are you ever modifying the IRP->Flags?

9

> In the case of undesired completion notification, does it get posted when

the I/O completes synchronously or with ERROR_IO_PENDING? Does it happen
for every type of request, or for a specific IOCTL code only?

Completion is completion. After several attempts to generate pathological
cases, I found that it seems that it is the IoCompleteRequest that
generates the event in the completion port. An I/O call that returns
ERROR_IO_PENDING means that IoCompleteRequest has not yet been called, but
will be called in the indefinite future (fine point: it means that at the
time you returned STATUS_PENDING and had done the IoMarkIrpPending, the
IRP had not yet been completed. However, race conditions can result in
the notification being placed in the completion port before the I/O call
has returned to its calling thread.

This kind of event is not even unusual.

I just ignore the return of the I/O call if it is TRUE, or FALSE &&
GetLastError() == ERROR_IO_PENDING). In either case, there will be a
completion port notification (assuming the driver is well-behaved and does
not actually fail to complete IRPs). If it is any other error, then the
IRP was rejected and there will be no IOCP notification. I don’t know if
I managed to get all the timings right, but I think this is what I have
observed.

If it isn’t right, I’d like to know the Whole Truth (and the Truth will
*not* make me free; I still charge the same amount!)
joe

NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

10

@Joe:

The stray completion port notification creates a race condition, where the caller might have already destroyed the OVERLAPPED block at the time of the completion thread picks it.

@Tim:

I suppose the device handle and the completion port are not inheritable?
Also: How soon do you close the event handle after GetOverlappedResult returns on completion?

11

> cases, I found that it seems that it is the IoCompleteRequest that

generates the event in the completion port.

Exactly: IopCompleteRequest calls KeInsertQueue, which puts the IRP to the completion port queue.

–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

12

Mr. Roberts: You observe thus ONLY on Win 7, not before and not after? Any specific SP?

PeterGV
OSR
@OSRDrivers

13

xxxxx@osr.com wrote:

Mr. Roberts: You observe thus ONLY on Win 7, not before and not after? Any specific SP?

It appears that the proper response here is “never mind”. The client
apparently added their own code that bypassed my carefully constructed APIs.

I appreciate all of your thoughtful speculation.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

14

The IO manager owner will be relieved :slight_smile:

-p

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Tim Roberts
Sent: Tuesday, May 13, 2014 10:11 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Completion Port Bypass

xxxxx@osr.com wrote:

Mr. Roberts: You observe thus ONLY on Win 7, not before and not after? Any specific SP?

It appears that the proper response here is “never mind”. The client apparently added their own code that bypassed my carefully constructed APIs.

I appreciate all of your thoughtful speculation.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer