HID minidriver: question for async I/O and surprise removal on Win8.1/10

(HID minidriver: question for async I/O and surprise removal on Win8.1/10)

Hello.

I have a HID minidriver (over USB), and I see a strange problem under Windows 8.1/10, but not Windows 7/Vista. My user-mode component queues an asynchronous write IRP to my minidriver (queued in a IOCSQ), just so that if the app dies, I can be notified and clean up in kernel-mode, and place the device in a reset state for its next use. Normally, should the device be unplugged from its USB port, I do expect that even if there is asynchronous I/O pending for the device that I will receive in the minidriver a surprise removal IRP so I can cancel outstanding I/O and cleanup as mandated for this IRP, and that the user-mode component be notified of device departure. But this is not the case under Windows 8.1 and 10 (no surprise removal IRP received), though it works as I expect it to under Windows 7/Vista. For the Windows 8.1/10 case, I receive no surprise removal IRP, and my user-mode component receives no notification for device departure (so it can close its handle), while for Windows 7/Vista I do get surprise removal, in which I cancel outstanding I/O and cleanup as mandated in kernel-mode, and my user-mode component receives a device departure message, so it can close its handle. Now I have a temporary workaround in my kernel-mode minidriver where I note the absence of the device, and clean up outstanding I/O. I then immediately receive a surprise removal IRP in my minidriver, and everything proceeds normally.
Does anyone have any idea why subsequent to a USB unplug that I must cancel outstanding asynchronous I/O before I may receive a surprise removal IRP, and that only in Windows 8.1/10? I did not realize that this is required.

Thank you,
Philip Lukidis

Philip,

I apologize for not having any advice for you, but I have a WDF UMDF2 hid minidriver that I want to get working with a USB device and it sounds like you at least have yours doing that. When you refer to user and kernel mode components are you referring to a UMDF minidriver as the user mode component and hidumdf.sys as the kernel mode component? If so then my driver should be similar in that it is a UMDF2 user mode minidriver (based on the vhidmini2 sample driver) and uses the system supplied mshidumdf.sys kernel mode driver for IO support. Assuming that I am correct would you be willing to post or send me your INF file so that I could see how you get your driver to install with a USB device? When my target device is not present my driver works fine as a virtual device that interacts with the user app written for that device, but when I have the device connected my driver is bypassed and the user app interacts directly with the device (or rather with the default system drivers: hidusb.sys, hidclass.sys and hidparse.sys that support it).

Randy

Hello.

I am writing again because I had received no response, and while I have a workaround, I am curious as to the cause of my problem, and would like a cleaner solution, if possible. Just to re-iterate: I have an HID minidriver, to which my user-mode component sends an output report, as an overlapped write (yes, CreateFile used FILE_FLAG_OVERLAPPED). Upon receiving the IRP, my HID minidriver queues it in an IOCSQ, and returns STATUS_PENDING. When I unplug my device On Windows 7/Vista, I receive as expected a surprise removal IRP, so I can cancel queued outstanding I/O (the pended and queued internal device I/O control IRP received via IOCTL_HID_SET_OUTPUT_REPORT/IOCTL_HID_WRITE_REPORT) and cleanup, and following this my user-mode component, which has registered for device departures, is notified, and can close its handle and do its own cleanup.

However, on Windows 8.1/10, unplugging the device does NOT result in my HID minidriver receiving a surprise removal IRP, and my user-mode component never receives a device departure notification. My workaround is to sense device departure in my completion routines (STATUS_NO_SUCH_DEVICE), and then cancel I/O pended I/O (and other cleanup possible at DISPATCH_LEVEL), whereupon I then get a surprise removal IRP in my HID minidriver, and everything then proceeds as under the Windows 7/Vista case.

My question is this: regardless if I have pended and queued I/O in my HID minidriver, should I not expect to get a surprise removal IRP in my HID minidriver after the device has been unplugged from USB? Checking over the documentation to IRP_MN_SURPRISE_REMOVAL, this would seem to be the case:

“Surprise remove (Windows 2000 and later versions of Windows)
On Windows 2000 and later systems, if a user unplugs a device from the machine without using the Unplug or Eject Hardware program, the PnP manager sends an IRP_MN_SURPRISE_REMOVAL IRP.
This case is called “surprise” removal because the drivers receive no advance warning.
In response to an IRP_MN_SURPRISE_REMOVAL IRP, the drivers for the device fail any outstanding I/O and release the hardware resources used by the device. The drivers must ensure that no components attempt to access the device because it is no longer present.”
https://msdn.microsoft.com/en-us/library/windows/hardware/ff564875(v=vs.85).aspx

Thanks for any help in clearing this up,
Philip Lukidis

Philip,

As in my previous comment I would like to know what you are doing in your hid minidriver to communicate with your device. I am now able to insert my driver into my device’s driver stack but attempts to communicate with the device have failed (although something gets sent, based on seeing USB activity when snooping the port, the device does not respond). My driver is a WDF UMDF2 hid minidriver based on the vhidmini2 sample model.

Randy

Hi Randy,

Sorry, by the time that I had checked and read your email, I had seen that Doron had answered you. My general setup is as follows: I have a user-mode DLL which writes/reads output/input HID reports and get/set feature (all specified in my device’s HID report descriptor). My kernel-mode component is an HID minidriver (WDM) over USB. Most of the device communication is done within internal device I/O control. You can find some documentation here for the IOCTLs which I service:
https://msdn.microsoft.com/en-us/library/windows/hardware/ff539926(v=vs.85).aspx

Upon reception of said IOCTLs, I forward them to my device, after some transformation, completion setup, and other vendor specific details (which would not apply to you).
HIDClass will filter our any requests which do not correlate with what is in the device’s HID report descriptor. Are you sure that your requests get passed down by HIDClass and actually reach your device? Did you use a USB bus analyzer to confirm this?

Thank you,
Philip Lukidis

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:bounce-603141-
xxxxx@lists.osr.com] On Behalf Of xxxxx@cox.net
Sent: February-25-16 2:38 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] HID minidriver: question for async I/O and surprise
removal on Win8.1/10

Philip,

As in my previous comment I would like to know what you are doing in your
hid minidriver to communicate with your device. I am now able to insert my
driver into my device’s driver stack but attempts to communicate with the
device have failed (although something gets sent, based on seeing USB
activity when snooping the port, the device does not respond). My driver is a
WDF UMDF2 hid minidriver based on the vhidmini2 sample model.

Randy


NTDEV is sponsored by OSR

Visit the list online at:
http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at
> http:</http:></http:></http:>

Philip Lukidis wrote:

…Just to re-iterate: I have an HID minidriver, to which my user-mode
component sends an output report, as an overlapped write (yes,
CreateFile used FILE_FLAG_OVERLAPPED).

The overlapped I/O part is irrelevant, of course. A kernel driver
neither knows nor cares about that.

However, on Windows 8.1/10, unplugging the device does NOT result in
my HID minidriver receiving a surprise removal IRP, and my user-mode
component never receives a device departure notification. My
workaround is to sense device departure in my completion routines
(STATUS_NO_SUCH_DEVICE), and then cancel I/O pended I/O (and other
cleanup possible at DISPATCH_LEVEL), whereupon I then get a surprise
removal IRP in my HID minidriver, and everything then proceeds as
under the Windows 7/Vista case.

It’s not supposed to work that way. The sequence goes like this. The
USB hub driver notices (through its interrupt requests) that a device
unplugged. It calls IoInvalidateDeviceRelations. PnP sends
IRP_MN_QUERY_DEVICE_RELATIONS to ask the hub what its children now are.
Your device won’t be listed, so PnP sends IRP_MN_SURPRISE_REMOVAL. That
should all happen asynchronously, regardless of outstanding requests.

Are you using KMDF? There’s no way you are blocking the PnP path in
some way, are you?

Doron is a HID expert. He might have extra knowledge about the handling
of surprise removal at your level, but I suspect it wasn’t supposed to
change between Win 7 and Win 8.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Philip,

My driver, based on the vhidmini2 sample model, is the UMDF dll component and it uses system supplied mshidumdf.sys as the kernel mode minidriver and WudfR as a lower filter driver/reflector. Since my driver is in user mode I can’t use internal device I/O. Mshidumdf.sys re-characterizes all IRP_MJ_XXXX codes into IRP_MJ_DEVICE_CONTROL before sending them to my driver so I can’t set dispatch events for IRP_MJ_READ or IRP_MJ_WRITE.

What functions/methods do you use to forward requests to your device?

I’m using WdfDeviceGetIoTarget to obtain a handle to the localIoTarget, WdfIoFormatRequestForWrite (with options set for Synchronous) and WdfRequestSend to forward a write request. I haven’t tried a read request yet because there would be nothing to read until I succeed at getting a write request to the device.

I don’t know what happens after I try to forward a request except that the WdfRequestSend function returns TRUE implying a request was sent and I do see USB activity to the device with a sniffer installed. I’m using SnoopyPro, an open source program, for this. The first dozen activities captured by the sniffer are the same as those I get for the device without my driver installed. These appear to be activities related to the system preparing to communicate with the device. After those first dozen activities, I see a series of BULK_OR_INTERRUPT_TRANSFER activities on EndPoints 0x81 (in) and 0x02 (out), when my driver is not installed, that show the data I’m sending and the devices replies to that data. With my driver installed there is no activity on endpoint 0x02 and the activity on 0x81 shows No Transfer Buffer. I don’t know what to do at this point.

Randy

Hi Tim,

Thanks for your reply. I’ve replied inline below.

The overlapped I/O part is irrelevant, of course. A kernel driver neither
knows nor cares about that.

Of course you’re right about that.

> However, on Windows 8.1/10, unplugging the device does NOT result in
> my HID minidriver receiving a surprise removal IRP, and my user-mode
> component never receives a device departure notification. My
> workaround is to sense device departure in my completion routines
> (STATUS_NO_SUCH_DEVICE), and then cancel I/O pended I/O (and other
> cleanup possible at DISPATCH_LEVEL), whereupon I then get a surprise
> removal IRP in my HID minidriver, and everything then proceeds as
> under the Windows 7/Vista case.

It’s not supposed to work that way. The sequence goes like this. The USB
hub driver notices (through its interrupt requests) that a device unplugged.
It calls IoInvalidateDeviceRelations. PnP sends
IRP_MN_QUERY_DEVICE_RELATIONS to ask the hub what its children now
are.
Your device won’t be listed, so PnP sends IRP_MN_SURPRISE_REMOVAL.
That should all happen asynchronously, regardless of outstanding requests.

That was my understanding as well.

Are you using KMDF? There’s no way you are blocking the PnP path in some
way, are you?

No, I’m using WDM. I don’t *believe* I’m blocking the PnP path, but I would have to double check this with the debugger. But even if I was blocking the PnP path, how would cancelling queued I/O have any impact, resulting in my getting the surprise removal IRP? I’m cancelling asynchronous I/O in a read report completion routine when I get STATUS_NO_SUCH_DEVICE. Just putting that code in place “unblocked” surprise removal. I’ll get that failing test machine and check with the debugger the moment I can.

Doron is a HID expert. He might have extra knowledge about the handling of
surprise removal at your level, but I suspect it wasn’t supposed to change
between Win 7 and Win 8.

I guess it does not change. I wonder what I could be doing to provoke this.

Hi Randy,

I’ve replied inline below.

My driver, based on the vhidmini2 sample model, is the UMDF dll component
and it uses system supplied mshidumdf.sys as the kernel mode minidriver
and WudfR as a lower filter driver/reflector. Since my driver is in user mode I
can’t use internal device I/O. Mshidumdf.sys re-characterizes all
IRP_MJ_XXXX codes into IRP_MJ_DEVICE_CONTROL before sending them to
my driver so I can’t set dispatch events for IRP_MJ_READ or IRP_MJ_WRITE.

What functions/methods do you use to forward requests to your device?

I use WDM, so it’s very different than your case. Being an HID minidriver, I receive
these IRPs from HIDClass, set my completion routine, and call IoCallDriver to
forward them to the next lower device object in the stack.

I’m using WdfDeviceGetIoTarget to obtain a handle to the localIoTarget,
WdfIoFormatRequestForWrite (with options set for Synchronous) and
WdfRequestSend to forward a write request. I haven’t tried a read request
yet because there would be nothing to read until I succeed at getting a write
request to the device.

For my case, read report requests are sent by HIDClass, while write report,
get/set feature are sent when my user-mode component makes the analogous calls.

I don’t know what happens after I try to forward a request except that the
WdfRequestSend function returns TRUE implying a request was sent and I do
see USB activity to the device with a sniffer installed. I’m using SnoopyPro, an
open source program, for this. The first dozen activities captured by the
sniffer are the same as those I get for the device without my driver installed.
These appear to be activities related to the system preparing to
communicate with the device. After those first dozen activities, I see a series
of BULK_OR_INTERRUPT_TRANSFER activities on EndPoints 0x81 (in) and
0x02 (out), when my driver is not installed, that show the data I’m sending
and the devices replies to that data. With my driver installed there is no
activity on endpoint 0x02 and the activity on 0x81 shows No Transfer Buffer. I
don’t know what to do at this point.

Randy

OK, if you see no data with your driver installed, then clearly no data reaches your
device for that case. I’m not sure why. Like I said before, if HIDClass thinks
that you have sent an invalid report not specified in your device’s HID report
descriptor, then it will filter it out, and your device will never get the data.
Perhaps this is what’s happening. Check to make sure that what you are
sending is valid as expressed in your device’s HID report descriptor.

Philip Lukidis

Thanks Philip, unfortunately as you have shown your situation is quite different from mine and I can’t apply any of what you are doing to my case. But I appreciate your feedback.

Randy

Philip Lukidis wrote:

> Are you using KMDF? There’s no way you are blocking the PnP path in some
> way, are you?
>
No, I’m using WDM. I don’t *believe* I’m blocking the PnP path, but I would have to double check this with the debugger. But even if I was blocking the PnP path, how would cancelling queued I/O have any impact, resulting in my getting the surprise removal IRP?

Imagine that you had an “outstanding requests” counter, and imagine that
you blocked PnP messages when that counter was non-zero. You wouldn’t
do that intentionally, but that’s the kind of thing I’m asking about.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Hi Tim, thanks for your reply; I’ve replied inline below

>> Are you using KMDF? There’s no way you are blocking the PnP path in
>> some way, are you?
>>
> No, I’m using WDM. I don’t *believe* I’m blocking the PnP path, but I
would have to double check this with the debugger. But even if I was
blocking the PnP path, how would cancelling queued I/O have any impact,
resulting in my getting the surprise removal IRP?

Imagine that you had an “outstanding requests” counter, and imagine that
you blocked PnP messages when that counter was non-zero. You wouldn’t
do that intentionally, but that’s the kind of thing I’m asking about.

Well, I don’t think that I’m blocking PnP, or if I can, I cannot see it. I could not attach
a kernel debugger to this machine at the moment, so I put in the manual crash on
demand registry value, and reproduced the issue by unplugging the device while
my user-mode client had a handle open and was communicating with the kernel-
mode driver. At this point, I gathered my traces, which indicated that I had
received two calls of IRP_MN_QUERY_DEVICE_RELATIONS, which I had simply
forwarded down the device stack. I saw no other PnP IRPs, so I manually provoked
a BSOD via the right control scrolllock-scrolllock key sequence, and looked at the
memory dump. I saw HIDClass blocking, but not my driver:

nt!KeWaitForSingleObject+0x375
HIDCLASS!HidpWaitForSignal+0x3f
HIDCLASS!HidpFdoPnp+0x3732
HIDCLASS!HidpIrpMajorPnp+0x6a
HIDCLASS!HidpMajorHandler+0x166

The only kernel stack from my driver was my kernel thread, waiting for requests,
but obviously not yet exited, as my minidriver had not yet received
IRP_MN_SURPRISE_REMOVAL.

Would anyone know why HIDCLASS would be so blocking? I do have my workaround,
But it would be nice to find out what’s going on.

Thank you,
Philip Lukidis

Philip Lukidis wrote:

The only kernel stack from my driver was my kernel thread, waiting for requests,
but obviously not yet exited, as my minidriver had not yet received
IRP_MN_SURPRISE_REMOVAL.

Why do you have a kernel thread? It’s not prohibited, of course, but I
have found very few circumstances where a dedicated kernel thread was
the right answer. A driver should spend most of its life responding to
requests from outside. You take action when you get a new IRP, or an
IRP completion, or an ISR, or a DPC, or a timer timoeuts, but the rest
of the time most drivers don’t need to be doing anything.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

> Why do you have a kernel thread? It’s not prohibited, of course, but I have

found very few circumstances where a dedicated kernel thread was the right
answer. A driver should spend most of its life responding to requests from
outside. You take action when you get a new IRP, or an IRP completion, or an
ISR, or a DPC, or a timer timoeuts, but the rest of the time most drivers don’t
need to be doing anything.

Hi Tim,

I don’t queue all requests, just those of a certain class. I’d rather not expand any
more, as it delves into vendor specific details.
But in any case, this is definitely not the source of the problem. My question is
why HIDCLASS is blocking in its PnP handler, and only unblocks after I cancel
queued I/O. This seems to occur in Windows 8.1 and higher. What could I do
to provoke this, if I don’t block in my PnP handler?

Thank you,
Philip Lukidis

It might be waiting for the ping pong IOCTL_READ_HID_REPORT irp it sent to the miniport to complete. I can try to look at the source later.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Philip Lukidis
Sent: Tuesday, March 1, 2016 1:52 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] HID minidriver: question for async I/O and surprise removal on Win8.1/10

> Why do you have a kernel thread? It’s not prohibited, of course, but
> I have found very few circumstances where a dedicated kernel thread
> was the right answer. A driver should spend most of its life
> responding to requests from outside. You take action when you get a
> new IRP, or an IRP completion, or an ISR, or a DPC, or a timer
> timoeuts, but the rest of the time most drivers don’t need to be doing anything.
>
Hi Tim,

I don’t queue all requests, just those of a certain class. I’d rather not expand any more, as it delves into vendor specific details.
But in any case, this is definitely not the source of the problem. My question is why HIDCLASS is blocking in its PnP handler, and only unblocks after I cancel queued I/O. This seems to occur in Windows 8.1 and higher. What could I do to provoke this, if I don’t block in my PnP handler?

Thank you,
Philip Lukidis


NTDEV is sponsored by OSR

Visit the list online at: https:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at https:

To unsubscribe, visit the List Server section of OSR Online at https:</https:></https:></https:>

> It might be waiting for the ping pong IOCTL_READ_HID_REPORT irp it sent to

the miniport to complete. I can try to look at the source later.

Thank you Doron, I would appreciate any hints. I should not be blocking/queueing
the read IRPs, and allow completion to continue. It’s peculiar that this issue only
occurs if there is queued I/O (output reports), and that I get the surprise removal IRP
in my minidriver only after having canceled queued I/O (output, never input reports).
Again, any help is much appreciated, Doron.

Philip Lukidis

I couldn’t quite tell what is going on from a cursory look.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Philip Lukidis
Sent: Tuesday, March 1, 2016 2:16 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] HID minidriver: question for async I/O and surprise removal on Win8.1/10

> It might be waiting for the ping pong IOCTL_READ_HID_REPORT irp it
> sent to the miniport to complete. I can try to look at the source later.
>
Thank you Doron, I would appreciate any hints. I should not be blocking/queueing the read IRPs, and allow completion to continue. It’s peculiar that this issue only occurs if there is queued I/O (output reports), and that I get the surprise removal IRP in my minidriver only after having canceled queued I/O (output, never input reports).
Again, any help is much appreciated, Doron.

Philip Lukidis


NTDEV is sponsored by OSR

Visit the list online at: https:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at https:

To unsubscribe, visit the List Server section of OSR Online at https:</https:></https:></https:>

This is a bug in hidclass. It is incorrectly blocking surprise remove until the IO (not all IO, only the ones that impact idle state) gets completed. Phillip is correct that it was introduced in Windows 8.1 when we re-factored the HIdclass. We will try to get it fixed in the next version of Windows. I will also check with the servicing folks to see if they would be willing to consider it for backporting. It is surprising that we never hit this so far.
Thanks,Vivek

From: xxxxx@microsoft.com
To: xxxxx@lists.osr.com
Subject: RE: [ntdev] HID minidriver: question for async I/O and surprise removal on Win8.1/10
Date: Wed, 2 Mar 2016 04:14:39 +0000

I couldn’t quite tell what is going on from a cursory look.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Philip Lukidis
Sent: Tuesday, March 1, 2016 2:16 PM
To: Windows System Software Devs Interest List
> Subject: RE: [ntdev] HID minidriver: question for async I/O and surprise removal on Win8.1/10
>
> > It might be waiting for the ping pong IOCTL_READ_HID_REPORT irp it
> > sent to the miniport to complete. I can try to look at the source later.
> >
> Thank you Doron, I would appreciate any hints. I should not be blocking/queueing the read IRPs, and allow completion to continue. It’s peculiar that this issue only occurs if there is queued I/O (output reports), and that I get the surprise removal IRP in my minidriver only after having canceled queued I/O (output, never input reports).
> Again, any help is much appreciated, Doron.
>
> Philip Lukidis
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: https:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at https:
>
> To unsubscribe, visit the List Server section of OSR Online at https:
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></https:></https:></https:>

Hi Vivek,

Thank you for the feedback, it’s much appreciated.

Philip Lukidis
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Vivek Gupta
Sent: March-01-16 11:35 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] HID minidriver: question for async I/O and surprise removal on Win8.1/10

This is a bug in hidclass. It is incorrectly blocking surprise remove until the IO (not all IO, only the ones that impact idle state) gets completed. Phillip is correct that it was introduced in Windows 8.1 when we re-factored the HIdclass. We will try to get it fixed in the next version of Windows. I will also check with the servicing folks to see if they would be willing to consider it for backporting. It is surprising that we never hit this so far.

Thanks,
Vivek

From: xxxxx@microsoft.commailto:xxxxx
> To: xxxxx@lists.osr.commailto:xxxxx
> Subject: RE: [ntdev] HID minidriver: question for async I/O and surprise removal on Win8.1/10
> Date: Wed, 2 Mar 2016 04:14:39 +0000
>
> I couldn’t quite tell what is going on from a cursory look.
>
> -----Original Message-----
> From: xxxxx@lists.osr.commailto:xxxxx [mailto:xxxxx@lists.osr.com] On Behalf Of Philip Lukidis
> Sent: Tuesday, March 1, 2016 2:16 PM
> To: Windows System Software Devs Interest List >
> Subject: RE: [ntdev] HID minidriver: question for async I/O and surprise removal on Win8.1/10
>
> > It might be waiting for the ping pong IOCTL_READ_HID_REPORT irp it
> > sent to the miniport to complete. I can try to look at the source later.
> >
> Thank you Doron, I would appreciate any hints. I should not be blocking/queueing the read IRPs, and allow completion to continue. It’s peculiar that this issue only occurs if there is queued I/O (output reports), and that I get the surprise removal IRP in my minidriver only after having canceled queued I/O (output, never input reports).
> Again, any help is much appreciated, Doron.
>
> Philip Lukidis
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: https:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at https:
>
> To unsubscribe, visit the List Server section of OSR Online at https:
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at http:


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></http:></http:></http:></https:></https:></https:></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx>