Quality of kernel drivers of well-known vendors like VMWare, McAfee, Juniper. Why does it suck so mu

Volodymyr_M_Shcherbyna-2 · October 27, 2010, 6:18pm

Hello everyone,

Just want to share some feedback I have regarding third party drivers I met
sometimes during my tests. Interesting to know, is it only happening to me ?
Or you also saw something simular when dealing with them?

Before releasing my code I always do heavy stress testing with
antiviruses/fws and I set verifier to verify mine and av/fw driver with
maximum checks. Mine driver is a typical TDI filter with some additional
functionality (I set PsLoadImageNotify callbacks to track process creation
events). (please, no need to point that TDI is depricated, I know that, I am
already taking care of it. This is not the purpose of this post)

So, testing with firewalls I always get BSODs with McAfee, Trend Micro (with
Trend Micro I have less problems, though) in their TDI filters. Even if I do
tests on clean machine (i.e., clean install of XP or Vista and there is NO
my driver) McAfee BSOD’s with pool corruption assert within 10 minutes of
stress testing.

(I was also troubleshooting issues with Juniper VPN client which is btw a
TDI filter. You enable verifier, run it, and you got immidiate BSOD. IIRC it
was related to not marking IRP pending, or something simular.)

Mine stress test tools are simple: create a lot of processes and make a lot
of connections, and it just dies :). I’ve sent request to McAfee about this
problem, but they never actually resolved it. But this is not the point, I
am interested to know your opinion about quality of those guys.

Today I was experimenting with my VmWare Workstation at home in Windows 7
x64 and I got a BSOD when doing verification of vmx86.sys driver. The test
case is very simple:

Fire verifier and set maximum checks for vmx86.sys
Reboot
Start any x86 virtual machine, and voila, you have the following:

DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught. This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this
driver.
If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA
will
be among the most commonly seen crashes.
Arguments:
Arg1: 00000000000000f6, Referencing user handle as KernelMode.
Arg2: 00000000000002e0, Handle value being referenced.
Arg3: fffffa8005b9fb30, Address of the current process.
Arg4: fffff880062da398, Address inside the driver that is performing the
incorrect reference.

Debugging Details:

BUGCHECK_STR: 0xc4_f6

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: VERIFIER_ENABLED_VISTA_MINIDUMP

PROCESS_NAME: vmware-vmx.exe

CURRENT_IRQL: 0

LAST_CONTROL_TRANSFER: from fffff800035493dc to fffff800030bf740

STACK_TEXT:
fffff88007953f48 fffff800035493dc : 00000000000000c4 00000000000000f6
00000000000002e0 fffffa8005b9fb30 : nt!KeBugCheckEx
fffff88007953f50 fffff8000355eae4 : 00000000000002e0 fffffa8005b9fb30
0000000000000007 fffff78000001000 : nt!VerifierBugCheckIfAppropriate+0x3c
fffff88007953f90 fffff8000331ab40 : fffff6fc00019ae8 fffff880079541f0
fffffa8004619500 fffffa800c0a2cc8 : nt!VfCheckUserHandle+0x1b4
fffff88007954070 fffff800033b8ab5 : 0000000000000000 0000000000000000
0000000000000000 0000007fffffff00 : nt! ?? ::NNGAKEGL::string'+0x20e2e fffff88007954140 fffff800033bde4d : fffffa800c0a2b10 fffff880079542a0 fffff6fb00000040 fffffa8004619510 : nt!ObpLookupObjectName+0x1b5 fffff88007954240 fffff8000335d654 : fffff880079544c8 0000000000000000 fffff88007954300 fffff800030bd93d : nt!ObOpenObjectByName+0x1cd fffff880079542f0 fffff8000335d72e : fffff88007954760 fffff880000f003f fffff88007954780 fffff8000335d700 : nt!CmCreateKey+0x2e1 fffff88007954460 fffff800030be993 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!NtCreateKey+0x2e fffff880079544b0 fffff800030baf30 : fffff8000354fb49 fffff88007954780 fffff880079548a0 0000000000261b00 : nt!KiSystemServiceCopyEnd+0x13 fffff880079546b8 fffff8000354fb49 : fffff88007954780 fffff880079548a0 0000000000261b00 fffff880062e5590 : nt!KiServiceLinkage fffff880079546c0 fffff880062da398 : 0000000000000000 fffff880062e5590 0000000000261b00 fffff880079548a0 : nt!VfZwCreateKey+0x99 fffff88007954720 0000000000000000 : fffff880062e5590 0000000000261b00 fffff880079548a0 00000000`00000000 : vmx86+0x3398

STACK_COMMAND: kb

FOLLOWUP_IP:
vmx86+3398
fffff880`062da398 413bc4 cmp eax,r12d

SYMBOL_STACK_INDEX: b

SYMBOL_NAME: vmx86+3398

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: vmx86

IMAGE_NAME: vmx86.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4a85fa8c

FAILURE_BUCKET_ID: X64_0xc4_f6_VRF_vmx86+3398

BUCKET_ID: X64_0xc4_f6_VRF_vmx86+3398

Followup: MachineOwner

This is not the first bug check I have, I also had something related to
memory corruption.

Is it a fault of verifier? I doubt. On the other hand, these companies
(McAfee, VmWare, Juniper) earn millions and spend millions on R&D teams, how
is it possible that they just never do Driver Verifier tests? (At least) …

Hope to hear some feedback on this : )

–
Volodymyr

Gary_Little-3 · October 27, 2010, 6:39pm

Never used Trend, but I have seen the same problems with Norton, which is
the worse, and MacAfee. Norton I will not even allow on any of my personal
machines. What I do use, and what has not failed with the DV settings I use
for testing my WFP driver, has been ESET’s products. This isn’t an ad for
ESET, simply a declaration of what I have observed.

Gary G. Little
H (952) 223-1349
C (952) 454-4629
xxxxx@comcast.net

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Volodymyr M.
Shcherbyna
Sent: Wednesday, October 27, 2010 5:18 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Quality of kernel drivers of well-known vendors like
VMWare, McAfee, Juniper. Why does it suck so much?

Hello everyone,

Just want to share some feedback I have regarding third party drivers I met
sometimes during my tests. Interesting to know, is it only happening to me ?

Or you also saw something simular when dealing with them?

Before releasing my code I always do heavy stress testing with
antiviruses/fws and I set verifier to verify mine and av/fw driver with
maximum checks. Mine driver is a typical TDI filter with some additional
functionality (I set PsLoadImageNotify callbacks to track process creation
events). (please, no need to point that TDI is depricated, I know that, I am
already taking care of it. This is not the purpose of this post)

So, testing with firewalls I always get BSODs with McAfee, Trend Micro (with
Trend Micro I have less problems, though) in their TDI filters. Even if I do
tests on clean machine (i.e., clean install of XP or Vista and there is NO
my driver) McAfee BSOD’s with pool corruption assert within 10 minutes of
stress testing.

(I was also troubleshooting issues with Juniper VPN client which is btw a
TDI filter. You enable verifier, run it, and you got immidiate BSOD. IIRC it
was related to not marking IRP pending, or something simular.)

Mine stress test tools are simple: create a lot of processes and make a lot
of connections, and it just dies :). I’ve sent request to McAfee about this
problem, but they never actually resolved it. But this is not the point, I
am interested to know your opinion about quality of those guys.

Today I was experimenting with my VmWare Workstation at home in Windows 7
x64 and I got a BSOD when doing verification of vmx86.sys driver. The test
case is very simple:

Fire verifier and set maximum checks for vmx86.sys
Reboot
Start any x86 virtual machine, and voila, you have the following:

DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught. This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this
driver.
If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA
will be among the most commonly seen crashes.
Arguments:
Arg1: 00000000000000f6, Referencing user handle as KernelMode.
Arg2: 00000000000002e0, Handle value being referenced.
Arg3: fffffa8005b9fb30, Address of the current process.
Arg4: fffff880062da398, Address inside the driver that is performing the
incorrect reference.

Debugging Details:

BUGCHECK_STR: 0xc4_f6

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: VERIFIER_ENABLED_VISTA_MINIDUMP

PROCESS_NAME: vmware-vmx.exe

CURRENT_IRQL: 0

LAST_CONTROL_TRANSFER: from fffff800035493dc to fffff800030bf740

STACK_TEXT:
fffff88007953f48 fffff800035493dc : 00000000000000c4 00000000000000f6
00000000000002e0 fffffa8005b9fb30 : nt!KeBugCheckEx
fffff88007953f50 fffff8000355eae4 : 00000000000002e0 fffffa8005b9fb30
0000000000000007 fffff78000001000 : nt!VerifierBugCheckIfAppropriate+0x3c
fffff88007953f90 fffff8000331ab40 : fffff6fc00019ae8 fffff880079541f0
fffffa8004619500 fffffa800c0a2cc8 : nt!VfCheckUserHandle+0x1b4
fffff88007954070 fffff800033b8ab5 : 0000000000000000 0000000000000000
0000000000000000 0000007fffffff00 : nt! ?? ::NNGAKEGL::string'+0x20e2e fffff88007954140 fffff800033bde4d : fffffa800c0a2b10 fffff880079542a0 fffff6fb00000040 fffffa8004619510 : nt!ObpLookupObjectName+0x1b5 fffff88007954240 fffff8000335d654 : fffff880079544c8 0000000000000000 fffff88007954300 fffff800030bd93d : nt!ObOpenObjectByName+0x1cd fffff880079542f0 fffff8000335d72e : fffff88007954760 fffff880000f003f fffff88007954780 fffff8000335d700 : nt!CmCreateKey+0x2e1 fffff88007954460 fffff800030be993 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!NtCreateKey+0x2e fffff880079544b0 fffff800030baf30 : fffff8000354fb49 fffff88007954780 fffff880079548a0 0000000000261b00 : nt!KiSystemServiceCopyEnd+0x13 fffff880079546b8 fffff8000354fb49 : fffff88007954780 fffff880079548a0 0000000000261b00 fffff880062e5590 : nt!KiServiceLinkage fffff880079546c0 fffff880062da398 : 0000000000000000 fffff880062e5590 0000000000261b00 fffff880079548a0 : nt!VfZwCreateKey+0x99 fffff88007954720 0000000000000000 : fffff880062e5590 0000000000261b00 fffff880079548a0 00000000`00000000 : vmx86+0x3398

STACK_COMMAND: kb

FOLLOWUP_IP:
vmx86+3398
fffff880`062da398 413bc4 cmp eax,r12d

SYMBOL_STACK_INDEX: b

SYMBOL_NAME: vmx86+3398

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: vmx86

IMAGE_NAME: vmx86.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4a85fa8c

FAILURE_BUCKET_ID: X64_0xc4_f6_VRF_vmx86+3398

BUCKET_ID: X64_0xc4_f6_VRF_vmx86+3398

Followup: MachineOwner

This is not the first bug check I have, I also had something related to
memory corruption.

Is it a fault of verifier? I doubt. On the other hand, these companies
(McAfee, VmWare, Juniper) earn millions and spend millions on R&D teams, how
is it possible that they just never do Driver Verifier tests? (At least) …

Hope to hear some feedback on this : )

–
Volodymyr

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

__________ Information from ESET Smart Security, version of virus signature
database 5569 (20101027) __________

The message was checked by ESET Smart Security.

http://www.eset.com

Alex_Grig · October 27, 2010, 6:47pm

This is why I don’t use antiviruses on my home PC. Proper security config is enough. The less 3rd party shit the better.

mm1 · October 27, 2010, 6:51pm

+1

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@broadcom.com
Sent: Wednesday, October 27, 2010 6:49 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Quality of kernel drivers of well-known vendors like
VMWare, McAfee, Juniper. Why does it suck so much?

This is why I don’t use antiviruses on my home PC. Proper security config is
enough. The less 3rd party shit the better.

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Bill_Wandel · October 27, 2010, 6:56pm

One of the VMWare drivers that is installed in the VM also has problems with
the checked kernel. I think that it was the VMWare video driver. The OS was
either W2K8 or W2K8 R2. It was asserts while booting.

Bill Wandel

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Volodymyr M.
Shcherbyna
Sent: Wednesday, October 27, 2010 6:18 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Quality of kernel drivers of well-known vendors like
VMWare, McAfee, Juniper. Why does it suck so much?

Hello everyone,

Just want to share some feedback I have regarding third party drivers I met
sometimes during my tests. Interesting to know, is it only happening to me ?

Or you also saw something simular when dealing with them?

Before releasing my code I always do heavy stress testing with
antiviruses/fws and I set verifier to verify mine and av/fw driver with
maximum checks. Mine driver is a typical TDI filter with some additional
functionality (I set PsLoadImageNotify callbacks to track process creation
events). (please, no need to point that TDI is depricated, I know that, I am
already taking care of it. This is not the purpose of this post)

So, testing with firewalls I always get BSODs with McAfee, Trend Micro (with
Trend Micro I have less problems, though) in their TDI filters. Even if I do
tests on clean machine (i.e., clean install of XP or Vista and there is NO
my driver) McAfee BSOD’s with pool corruption assert within 10 minutes of
stress testing.

(I was also troubleshooting issues with Juniper VPN client which is btw a
TDI filter. You enable verifier, run it, and you got immidiate BSOD. IIRC it
was related to not marking IRP pending, or something simular.)

Mine stress test tools are simple: create a lot of processes and make a lot
of connections, and it just dies :). I’ve sent request to McAfee about this
problem, but they never actually resolved it. But this is not the point, I
am interested to know your opinion about quality of those guys.

Today I was experimenting with my VmWare Workstation at home in Windows 7
x64 and I got a BSOD when doing verification of vmx86.sys driver. The test
case is very simple:

Fire verifier and set maximum checks for vmx86.sys
Reboot
Start any x86 virtual machine, and voila, you have the following:

DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught. This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this
driver.
If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA
will be among the most commonly seen crashes.
Arguments:
Arg1: 00000000000000f6, Referencing user handle as KernelMode.
Arg2: 00000000000002e0, Handle value being referenced.
Arg3: fffffa8005b9fb30, Address of the current process.
Arg4: fffff880062da398, Address inside the driver that is performing the
incorrect reference.

Debugging Details:

BUGCHECK_STR: 0xc4_f6

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: VERIFIER_ENABLED_VISTA_MINIDUMP

PROCESS_NAME: vmware-vmx.exe

CURRENT_IRQL: 0

LAST_CONTROL_TRANSFER: from fffff800035493dc to fffff800030bf740

STACK_TEXT:
fffff88007953f48 fffff800035493dc : 00000000000000c4 00000000000000f6
00000000000002e0 fffffa8005b9fb30 : nt!KeBugCheckEx
fffff88007953f50 fffff8000355eae4 : 00000000000002e0 fffffa8005b9fb30
0000000000000007 fffff78000001000 : nt!VerifierBugCheckIfAppropriate+0x3c
fffff88007953f90 fffff8000331ab40 : fffff6fc00019ae8 fffff880079541f0
fffffa8004619500 fffffa800c0a2cc8 : nt!VfCheckUserHandle+0x1b4
fffff88007954070 fffff800033b8ab5 : 0000000000000000 0000000000000000
0000000000000000 0000007fffffff00 : nt! ?? ::NNGAKEGL::string'+0x20e2e fffff88007954140 fffff800033bde4d : fffffa800c0a2b10 fffff880079542a0 fffff6fb00000040 fffffa8004619510 : nt!ObpLookupObjectName+0x1b5 fffff88007954240 fffff8000335d654 : fffff880079544c8 0000000000000000 fffff88007954300 fffff800030bd93d : nt!ObOpenObjectByName+0x1cd fffff880079542f0 fffff8000335d72e : fffff88007954760 fffff880000f003f fffff88007954780 fffff8000335d700 : nt!CmCreateKey+0x2e1 fffff88007954460 fffff800030be993 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!NtCreateKey+0x2e fffff880079544b0 fffff800030baf30 : fffff8000354fb49 fffff88007954780 fffff880079548a0 0000000000261b00 : nt!KiSystemServiceCopyEnd+0x13 fffff880079546b8 fffff8000354fb49 : fffff88007954780 fffff880079548a0 0000000000261b00 fffff880062e5590 : nt!KiServiceLinkage fffff880079546c0 fffff880062da398 : 0000000000000000 fffff880062e5590 0000000000261b00 fffff880079548a0 : nt!VfZwCreateKey+0x99 fffff88007954720 0000000000000000 : fffff880062e5590 0000000000261b00 fffff880079548a0 00000000`00000000 : vmx86+0x3398

STACK_COMMAND: kb

FOLLOWUP_IP:
vmx86+3398
fffff880`062da398 413bc4 cmp eax,r12d

SYMBOL_STACK_INDEX: b

SYMBOL_NAME: vmx86+3398

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: vmx86

IMAGE_NAME: vmx86.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4a85fa8c

FAILURE_BUCKET_ID: X64_0xc4_f6_VRF_vmx86+3398

BUCKET_ID: X64_0xc4_f6_VRF_vmx86+3398

Followup: MachineOwner

This is not the first bug check I have, I also had something related to
memory corruption.

Is it a fault of verifier? I doubt. On the other hand, these companies
(McAfee, VmWare, Juniper) earn millions and spend millions on R&D teams, how
is it possible that they just never do Driver Verifier tests? (At least) …

Hope to hear some feedback on this : )

–
Volodymyr

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

mm1 · October 27, 2010, 7:06pm

>On the other hand, these companies (McAfee, VmWare, Juniper) earn millions
and spend millions on R&D teams, how is it possible that they just never do
Driver Verifier tests? (At least) …

Although Driver Verifier has been around since W2K (I think), it certainly
hasn’t been in mainstream usage anywhere near that long, in my opinion, and
if you’re considering PREFast as well, that’s far newer and in my experience
unfortunately not at all commonly used.

While I hear what you’re saying in general, I think that it’s important to
observe that there are many ways to go about debugging, and stable products
have definitely been built with the use of either Driver Verifier and/or
PREFast.

For example, I use VMWare workstation every day, and while it certainly has
some issues like all software, I would call it a stable, at least for my
purposes. I’m not saying that that error you mentioned below isn’t a
problem (I have no idea), but in the presence of limited resources to throw
at all problems, making code compliant with tools after the fact
(hypothetically) can be hard to justify (and is just not possible in some
cases) if everything is in working order in practice and you have other fish
to fry on top of it.

This is in my opinion far less of a problem with verifier than it is with
PREFast, where it can be painfully acute how large the cost of getting old
code to be prefast clean can be (or lint, et. c.), not to mention that as
with any tools, there’s a cost for using it in the first place. In the case
of prefast, it’s essentially unreadable code, which gets into
cultural/religious coding issues, which are also real and important.

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Volodymyr M.
Shcherbyna
Sent: Wednesday, October 27, 2010 6:18 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Quality of kernel drivers of well-known vendors like
VMWare, McAfee, Juniper. Why does it suck so much?

Hello everyone,

Just want to share some feedback I have regarding third party drivers I met
sometimes during my tests. Interesting to know, is it only happening to me ?

Or you also saw something simular when dealing with them?

Before releasing my code I always do heavy stress testing with
antiviruses/fws and I set verifier to verify mine and av/fw driver with
maximum checks. Mine driver is a typical TDI filter with some additional
functionality (I set PsLoadImageNotify callbacks to track process creation
events). (please, no need to point that TDI is depricated, I know that, I am

already taking care of it. This is not the purpose of this post)

So, testing with firewalls I always get BSODs with McAfee, Trend Micro (with

Trend Micro I have less problems, though) in their TDI filters. Even if I do

tests on clean machine (i.e., clean install of XP or Vista and there is NO
my driver) McAfee BSOD’s with pool corruption assert within 10 minutes of
stress testing.

(I was also troubleshooting issues with Juniper VPN client which is btw a
TDI filter. You enable verifier, run it, and you got immidiate BSOD. IIRC it

was related to not marking IRP pending, or something simular.)

Mine stress test tools are simple: create a lot of processes and make a lot
of connections, and it just dies :). I’ve sent request to McAfee about this
problem, but they never actually resolved it. But this is not the point, I
am interested to know your opinion about quality of those guys.

Today I was experimenting with my VmWare Workstation at home in Windows 7
x64 and I got a BSOD when doing verification of vmx86.sys driver. The test
case is very simple:

Fire verifier and set maximum checks for vmx86.sys
Reboot
Start any x86 virtual machine, and voila, you have the following:

DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught. This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this
driver.
If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA
will
be among the most commonly seen crashes.
Arguments:
Arg1: 00000000000000f6, Referencing user handle as KernelMode.
Arg2: 00000000000002e0, Handle value being referenced.
Arg3: fffffa8005b9fb30, Address of the current process.
Arg4: fffff880062da398, Address inside the driver that is performing the
incorrect reference.

Debugging Details:

BUGCHECK_STR: 0xc4_f6

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: VERIFIER_ENABLED_VISTA_MINIDUMP

PROCESS_NAME: vmware-vmx.exe

CURRENT_IRQL: 0

LAST_CONTROL_TRANSFER: from fffff800035493dc to fffff800030bf740

STACK_TEXT:
fffff88007953f48 fffff800035493dc : 00000000000000c4 00000000000000f6
00000000000002e0 fffffa8005b9fb30 : nt!KeBugCheckEx
fffff88007953f50 fffff8000355eae4 : 00000000000002e0 fffffa8005b9fb30
0000000000000007 fffff78000001000 : nt!VerifierBugCheckIfAppropriate+0x3c
fffff88007953f90 fffff8000331ab40 : fffff6fc00019ae8 fffff880079541f0
fffffa8004619500 fffffa800c0a2cc8 : nt!VfCheckUserHandle+0x1b4
fffff88007954070 fffff800033b8ab5 : 0000000000000000 0000000000000000
0000000000000000 0000007fffffff00 : nt! ?? ::NNGAKEGL::string'+0x20e2e fffff88007954140 fffff800033bde4d : fffffa800c0a2b10 fffff880079542a0 fffff6fb00000040 fffffa8004619510 : nt!ObpLookupObjectName+0x1b5 fffff88007954240 fffff8000335d654 : fffff880079544c8 0000000000000000 fffff88007954300 fffff800030bd93d : nt!ObOpenObjectByName+0x1cd fffff880079542f0 fffff8000335d72e : fffff88007954760 fffff880000f003f fffff88007954780 fffff8000335d700 : nt!CmCreateKey+0x2e1 fffff88007954460 fffff800030be993 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!NtCreateKey+0x2e fffff880079544b0 fffff800030baf30 : fffff8000354fb49 fffff88007954780 fffff880079548a0 0000000000261b00 : nt!KiSystemServiceCopyEnd+0x13 fffff880079546b8 fffff8000354fb49 : fffff88007954780 fffff880079548a0 0000000000261b00 fffff880062e5590 : nt!KiServiceLinkage fffff880079546c0 fffff880062da398 : 0000000000000000 fffff880062e5590 0000000000261b00 fffff880079548a0 : nt!VfZwCreateKey+0x99 fffff88007954720 0000000000000000 : fffff880062e5590 0000000000261b00 fffff880079548a0 00000000`00000000 : vmx86+0x3398

STACK_COMMAND: kb

FOLLOWUP_IP:
vmx86+3398
fffff880`062da398 413bc4 cmp eax,r12d

SYMBOL_STACK_INDEX: b

SYMBOL_NAME: vmx86+3398

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: vmx86

IMAGE_NAME: vmx86.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4a85fa8c

FAILURE_BUCKET_ID: X64_0xc4_f6_VRF_vmx86+3398

BUCKET_ID: X64_0xc4_f6_VRF_vmx86+3398

Followup: MachineOwner

This is not the first bug check I have, I also had something related to
memory corruption.

Is it a fault of verifier? I doubt. On the other hand, these companies
(McAfee, VmWare, Juniper) earn millions and spend millions on R&D teams, how

is it possible that they just never do Driver Verifier tests? (At least) …

Hope to hear some feedback on this : )

–
Volodymyr

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Rossetoecioccolato · October 27, 2010, 7:22pm

Bull-$@#! The driver verifier is a part of reasonable due care. And I am
personally thankful for it. It has saved me a lot of headaches down the
road. We develop with the verifier (and Prefast) ON.

Regards,

George.

“Martin O’Brien” wrote in message
news:xxxxx@ntdev…
> >On the other hand, these companies (McAfee, VmWare, Juniper) earn
> >millions
> and spend millions on R&D teams, how is it possible that they just never
> do
> Driver Verifier tests? (At least) …
>
> Although Driver Verifier has been around since W2K (I think), it certainly
> hasn’t been in mainstream usage anywhere near that long, in my opinion,
> and
> if you’re considering PREFast as well, that’s far newer and in my
> experience
> unfortunately not at all commonly used.
>
> While I hear what you’re saying in general, I think that it’s important to
> observe that there are many ways to go about debugging, and stable
> products
> have definitely been built with the use of either Driver Verifier and/or
> PREFast.
>
> For example, I use VMWare workstation every day, and while it certainly
> has
> some issues like all software, I would call it a stable, at least for my
> purposes. I’m not saying that that error you mentioned below isn’t a
> problem (I have no idea), but in the presence of limited resources to
> throw
> at all problems, making code compliant with tools after the fact
> (hypothetically) can be hard to justify (and is just not possible in some
> cases) if everything is in working order in practice and you have other
> fish
> to fry on top of it.
>
> This is in my opinion far less of a problem with verifier than it is with
> PREFast, where it can be painfully acute how large the cost of getting old
> code to be prefast clean can be (or lint, et. c.), not to mention that as
> with any tools, there’s a cost for using it in the first place. In the
> case
> of prefast, it’s essentially unreadable code, which gets into
> cultural/religious coding issues, which are also real and important.
>
>
>
> mm
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Volodymyr M.
> Shcherbyna
> Sent: Wednesday, October 27, 2010 6:18 PM
> To: Windows System Software Devs Interest List
> Subject: [ntdev] Quality of kernel drivers of well-known vendors like
> VMWare, McAfee, Juniper. Why does it suck so much?
>
> Hello everyone,
>
> Just want to share some feedback I have regarding third party drivers I
> met
> sometimes during my tests. Interesting to know, is it only happening to me
> ?
>
> Or you also saw something simular when dealing with them?
>
> Before releasing my code I always do heavy stress testing with
> antiviruses/fws and I set verifier to verify mine and av/fw driver with
> maximum checks. Mine driver is a typical TDI filter with some additional
> functionality (I set PsLoadImageNotify callbacks to track process creation
> events). (please, no need to point that TDI is depricated, I know that, I
> am
>
> already taking care of it. This is not the purpose of this post)
>
> So, testing with firewalls I always get BSODs with McAfee, Trend Micro
> (with
>
> Trend Micro I have less problems, though) in their TDI filters. Even if I
> do
>
> tests on clean machine (i.e., clean install of XP or Vista and there is NO
> my driver) McAfee BSOD’s with pool corruption assert within 10 minutes of
> stress testing.
>
> (I was also troubleshooting issues with Juniper VPN client which is btw a
> TDI filter. You enable verifier, run it, and you got immidiate BSOD. IIRC
> it
>
> was related to not marking IRP pending, or something simular.)
>
> Mine stress test tools are simple: create a lot of processes and make a
> lot
> of connections, and it just dies :). I’ve sent request to McAfee about
> this
> problem, but they never actually resolved it. But this is not the point, I
> am interested to know your opinion about quality of those guys.
>
> Today I was experimenting with my VmWare Workstation at home in Windows 7
> x64 and I got a BSOD when doing verification of vmx86.sys driver. The test
> case is very simple:
>
> 1. Fire verifier and set maximum checks for vmx86.sys
>
> 2. Reboot
>
> 3. Start any x86 virtual machine, and voila, you have the following:
>
> DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
> A device driver attempting to corrupt the system has been caught. This is
> because the driver was specified in the registry as being suspect (by the
> administrator) and the kernel has enabled substantial checking of this
> driver.
> If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA
> will
> be among the most commonly seen crashes.
> Arguments:
> Arg1: 00000000000000f6, Referencing user handle as KernelMode.
> Arg2: 00000000000002e0, Handle value being referenced.
> Arg3: fffffa8005b9fb30, Address of the current process.
> Arg4: fffff880062da398, Address inside the driver that is performing the
> incorrect reference.
>
> Debugging Details:
> ------------------
>
>
> BUGCHECK_STR: 0xc4_f6
>
> CUSTOMER_CRASH_COUNT: 1
>
> DEFAULT_BUCKET_ID: VERIFIER_ENABLED_VISTA_MINIDUMP
>
> PROCESS_NAME: vmware-vmx.exe
>
> CURRENT_IRQL: 0
>
> LAST_CONTROL_TRANSFER: from fffff800035493dc to fffff800030bf740
>
> STACK_TEXT:
> fffff88007953f48 fffff800035493dc : 00000000000000c4 00000000000000f6
> 00000000000002e0 fffffa8005b9fb30 : nt!KeBugCheckEx
> fffff88007953f50 fffff8000355eae4 : 00000000000002e0 fffffa8005b9fb30
> 0000000000000007 fffff78000001000 :
> nt!VerifierBugCheckIfAppropriate+0x3c
> fffff88007953f90 fffff8000331ab40 : fffff6fc00019ae8 fffff880079541f0
> fffffa8004619500 fffffa800c0a2cc8 : nt!VfCheckUserHandle+0x1b4
> fffff88007954070 fffff800033b8ab5 : 0000000000000000 0000000000000000
> 0000000000000000 0000007fffffff00 : nt! ?? ::NNGAKEGL::string'+0x20e2e > fffff88007954140 fffff800033bde4d : fffffa800c0a2b10 fffff880079542a0 > fffff6fb00000040 fffffa8004619510 : nt!ObpLookupObjectName+0x1b5 > fffff88007954240 fffff8000335d654 : fffff880079544c8 0000000000000000 > fffff88007954300 fffff800030bd93d : nt!ObOpenObjectByName+0x1cd > fffff880079542f0 fffff8000335d72e : fffff88007954760 fffff880000f003f > fffff88007954780 fffff8000335d700 : nt!CmCreateKey+0x2e1 > fffff88007954460 fffff800030be993 : 0000000000000000 0000000000000000 > 0000000000000000 0000000000000000 : nt!NtCreateKey+0x2e > fffff880079544b0 fffff800030baf30 : fffff8000354fb49 fffff88007954780 > fffff880079548a0 0000000000261b00 : nt!KiSystemServiceCopyEnd+0x13 > fffff880079546b8 fffff8000354fb49 : fffff88007954780 fffff880079548a0 > 0000000000261b00 fffff880062e5590 : nt!KiServiceLinkage > fffff880079546c0 fffff880062da398 : 0000000000000000 fffff880062e5590 > 0000000000261b00 fffff880079548a0 : nt!VfZwCreateKey+0x99 > fffff88007954720 0000000000000000 : fffff880062e5590 0000000000261b00 > fffff880079548a0 0000000000000000 : vmx86+0x3398 > > > STACK_COMMAND: kb > > FOLLOWUP_IP: > vmx86+3398 > fffff880062da398 413bc4 cmp eax,r12d
>
> SYMBOL_STACK_INDEX: b
>
> SYMBOL_NAME: vmx86+3398
>
> FOLLOWUP_NAME: MachineOwner
>
> MODULE_NAME: vmx86
>
> IMAGE_NAME: vmx86.sys
>
> DEBUG_FLR_IMAGE_TIMESTAMP: 4a85fa8c
>
> FAILURE_BUCKET_ID: X64_0xc4_f6_VRF_vmx86+3398
>
> BUCKET_ID: X64_0xc4_f6_VRF_vmx86+3398
>
> Followup: MachineOwner
> ---------
>
> This is not the first bug check I have, I also had something related to
> memory corruption.
>
> Is it a fault of verifier? I doubt. On the other hand, these companies
> (McAfee, VmWare, Juniper) earn millions and spend millions on R&D teams,
> how
>
> is it possible that they just never do Driver Verifier tests? (At least)
> …
>
> Hope to hear some feedback on this : )
>
> –
> Volodymyr
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>

mm1 · October 27, 2010, 7:25pm

You’re saying that it doesn’t matter if the product works or not, unless
they run Driver Verifier, it’s crap?

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of George M. Garner Jr.
Sent: Wednesday, October 27, 2010 7:22 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Quality of kernel drivers of well-known vendors like
VMWare, McAfee, Juniper. Why does it suck so much?

Bull-$@#! The driver verifier is a part of reasonable due care. And I am
personally thankful for it. It has saved me a lot of headaches down the
road. We develop with the verifier (and Prefast) ON.

Regards,

George.

“Martin O’Brien” wrote in message
news:xxxxx@ntdev…
> >On the other hand, these companies (McAfee, VmWare, Juniper) earn
> >millions
> and spend millions on R&D teams, how is it possible that they just never
> do
> Driver Verifier tests? (At least) …
>
> Although Driver Verifier has been around since W2K (I think), it certainly
> hasn’t been in mainstream usage anywhere near that long, in my opinion,
> and
> if you’re considering PREFast as well, that’s far newer and in my
> experience
> unfortunately not at all commonly used.
>
> While I hear what you’re saying in general, I think that it’s important to
> observe that there are many ways to go about debugging, and stable
> products
> have definitely been built with the use of either Driver Verifier and/or
> PREFast.
>
> For example, I use VMWare workstation every day, and while it certainly
> has
> some issues like all software, I would call it a stable, at least for my
> purposes. I’m not saying that that error you mentioned below isn’t a
> problem (I have no idea), but in the presence of limited resources to
> throw
> at all problems, making code compliant with tools after the fact
> (hypothetically) can be hard to justify (and is just not possible in some
> cases) if everything is in working order in practice and you have other
> fish
> to fry on top of it.
>
> This is in my opinion far less of a problem with verifier than it is with
> PREFast, where it can be painfully acute how large the cost of getting old
> code to be prefast clean can be (or lint, et. c.), not to mention that as
> with any tools, there’s a cost for using it in the first place. In the
> case
> of prefast, it’s essentially unreadable code, which gets into
> cultural/religious coding issues, which are also real and important.
>
>
>
> mm
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Volodymyr M.
> Shcherbyna
> Sent: Wednesday, October 27, 2010 6:18 PM
> To: Windows System Software Devs Interest List
> Subject: [ntdev] Quality of kernel drivers of well-known vendors like
> VMWare, McAfee, Juniper. Why does it suck so much?
>
> Hello everyone,
>
> Just want to share some feedback I have regarding third party drivers I
> met
> sometimes during my tests. Interesting to know, is it only happening to me

> ?
>
> Or you also saw something simular when dealing with them?
>
> Before releasing my code I always do heavy stress testing with
> antiviruses/fws and I set verifier to verify mine and av/fw driver with
> maximum checks. Mine driver is a typical TDI filter with some additional
> functionality (I set PsLoadImageNotify callbacks to track process creation
> events). (please, no need to point that TDI is depricated, I know that, I
> am
>
> already taking care of it. This is not the purpose of this post)
>
> So, testing with firewalls I always get BSODs with McAfee, Trend Micro
> (with
>
> Trend Micro I have less problems, though) in their TDI filters. Even if I
> do
>
> tests on clean machine (i.e., clean install of XP or Vista and there is NO
> my driver) McAfee BSOD’s with pool corruption assert within 10 minutes of
> stress testing.
>
> (I was also troubleshooting issues with Juniper VPN client which is btw a
> TDI filter. You enable verifier, run it, and you got immidiate BSOD. IIRC
> it
>
> was related to not marking IRP pending, or something simular.)
>
> Mine stress test tools are simple: create a lot of processes and make a
> lot
> of connections, and it just dies :). I’ve sent request to McAfee about
> this
> problem, but they never actually resolved it. But this is not the point, I
> am interested to know your opinion about quality of those guys.
>
> Today I was experimenting with my VmWare Workstation at home in Windows 7
> x64 and I got a BSOD when doing verification of vmx86.sys driver. The test
> case is very simple:
>
> 1. Fire verifier and set maximum checks for vmx86.sys
>
> 2. Reboot
>
> 3. Start any x86 virtual machine, and voila, you have the following:
>
> DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
> A device driver attempting to corrupt the system has been caught. This is
> because the driver was specified in the registry as being suspect (by the
> administrator) and the kernel has enabled substantial checking of this
> driver.
> If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA
> will
> be among the most commonly seen crashes.
> Arguments:
> Arg1: 00000000000000f6, Referencing user handle as KernelMode.
> Arg2: 00000000000002e0, Handle value being referenced.
> Arg3: fffffa8005b9fb30, Address of the current process.
> Arg4: fffff880062da398, Address inside the driver that is performing the
> incorrect reference.
>
> Debugging Details:
> ------------------
>
>
> BUGCHECK_STR: 0xc4_f6
>
> CUSTOMER_CRASH_COUNT: 1
>
> DEFAULT_BUCKET_ID: VERIFIER_ENABLED_VISTA_MINIDUMP
>
> PROCESS_NAME: vmware-vmx.exe
>
> CURRENT_IRQL: 0
>
> LAST_CONTROL_TRANSFER: from fffff800035493dc to fffff800030bf740
>
> STACK_TEXT:
> fffff88007953f48 fffff800035493dc : 00000000000000c4 00000000000000f6
> 00000000000002e0 fffffa8005b9fb30 : nt!KeBugCheckEx
> fffff88007953f50 fffff8000355eae4 : 00000000000002e0 fffffa8005b9fb30
> 0000000000000007 fffff78000001000 :
> nt!VerifierBugCheckIfAppropriate+0x3c
> fffff88007953f90 fffff8000331ab40 : fffff6fc00019ae8 fffff880079541f0
> fffffa8004619500 fffffa800c0a2cc8 : nt!VfCheckUserHandle+0x1b4
> fffff88007954070 fffff800033b8ab5 : 0000000000000000 0000000000000000
> 0000000000000000 0000007fffffff00 : nt! ?? ::NNGAKEGL::string'+0x20e2e > fffff88007954140 fffff800033bde4d : fffffa800c0a2b10 fffff880079542a0 > fffff6fb00000040 fffffa8004619510 : nt!ObpLookupObjectName+0x1b5 > fffff88007954240 fffff8000335d654 : fffff880079544c8 0000000000000000 > fffff88007954300 fffff800030bd93d : nt!ObOpenObjectByName+0x1cd > fffff880079542f0 fffff8000335d72e : fffff88007954760 fffff880000f003f > fffff88007954780 fffff8000335d700 : nt!CmCreateKey+0x2e1 > fffff88007954460 fffff800030be993 : 0000000000000000 0000000000000000 > 0000000000000000 0000000000000000 : nt!NtCreateKey+0x2e > fffff880079544b0 fffff800030baf30 : fffff8000354fb49 fffff88007954780 > fffff880079548a0 0000000000261b00 : nt!KiSystemServiceCopyEnd+0x13 > fffff880079546b8 fffff8000354fb49 : fffff88007954780 fffff880079548a0 > 0000000000261b00 fffff880062e5590 : nt!KiServiceLinkage > fffff880079546c0 fffff880062da398 : 0000000000000000 fffff880062e5590 > 0000000000261b00 fffff880079548a0 : nt!VfZwCreateKey+0x99 > fffff88007954720 0000000000000000 : fffff880062e5590 0000000000261b00 > fffff880079548a0 0000000000000000 : vmx86+0x3398 > > > STACK_COMMAND: kb > > FOLLOWUP_IP: > vmx86+3398 > fffff880062da398 413bc4 cmp eax,r12d
>
> SYMBOL_STACK_INDEX: b
>
> SYMBOL_NAME: vmx86+3398
>
> FOLLOWUP_NAME: MachineOwner
>
> MODULE_NAME: vmx86
>
> IMAGE_NAME: vmx86.sys
>
> DEBUG_FLR_IMAGE_TIMESTAMP: 4a85fa8c
>
> FAILURE_BUCKET_ID: X64_0xc4_f6_VRF_vmx86+3398
>
> BUCKET_ID: X64_0xc4_f6_VRF_vmx86+3398
>
> Followup: MachineOwner
> ---------
>
> This is not the first bug check I have, I also had something related to
> memory corruption.
>
> Is it a fault of verifier? I doubt. On the other hand, these companies
> (McAfee, VmWare, Juniper) earn millions and spend millions on R&D teams,
> how
>
> is it possible that they just never do Driver Verifier tests? (At least)
> …
>
> Hope to hear some feedback on this : )
>
> –
> Volodymyr
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Rossetoecioccolato · October 27, 2010, 7:45pm

Martin,

I am saying:

When the driver verifier bugchecks there is usually a good reason for it;
and
The most insidious bugs have no apparent effects until one day random
data corruption hits the right spot and the computer orders a drone to fire
a missile… And then you end up the lead story on the nightly news.

The fact that some buggy driver appears to “work” is no indication of what
might happen down the road.

The driver verifier can be a pain in the ass at times. However, a pain in
the ass during development is better than a pain in the ass down the road.
Just my 2 centimes worth.

Regards,

George.

“Martin O’Brien” wrote in message
news:xxxxx@ntdev…
> You’re saying that it doesn’t matter if the product works or not, unless
> they run Driver Verifier, it’s crap?
>
>
> mm
>
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of George M. Garner
> Jr.
> Sent: Wednesday, October 27, 2010 7:22 PM
> To: Windows System Software Devs Interest List
> Subject: Re:[ntdev] Quality of kernel drivers of well-known vendors like
> VMWare, McAfee, Juniper. Why does it suck so much?
>
> Bull-$@#! The driver verifier is a part of reasonable due care. And I am
> personally thankful for it. It has saved me a lot of headaches down the
> road. We develop with the verifier (and Prefast) ON.
>
> Regards,
>
> George.
>
> “Martin O’Brien” wrote in message
> news:xxxxx@ntdev…
>> >On the other hand, these companies (McAfee, VmWare, Juniper) earn
>> >millions
>> and spend millions on R&D teams, how is it possible that they just never
>> do
>> Driver Verifier tests? (At least) …
>>
>> Although Driver Verifier has been around since W2K (I think), it
>> certainly
>> hasn’t been in mainstream usage anywhere near that long, in my opinion,
>> and
>> if you’re considering PREFast as well, that’s far newer and in my
>> experience
>> unfortunately not at all commonly used.
>>
>> While I hear what you’re saying in general, I think that it’s important
>> to
>> observe that there are many ways to go about debugging, and stable
>> products
>> have definitely been built with the use of either Driver Verifier and/or
>> PREFast.
>>
>> For example, I use VMWare workstation every day, and while it certainly
>> has
>> some issues like all software, I would call it a stable, at least for my
>> purposes. I’m not saying that that error you mentioned below isn’t a
>> problem (I have no idea), but in the presence of limited resources to
>> throw
>> at all problems, making code compliant with tools after the fact
>> (hypothetically) can be hard to justify (and is just not possible in some
>> cases) if everything is in working order in practice and you have other
>> fish
>> to fry on top of it.
>>
>> This is in my opinion far less of a problem with verifier than it is with
>> PREFast, where it can be painfully acute how large the cost of getting
>> old
>> code to be prefast clean can be (or lint, et. c.), not to mention that as
>> with any tools, there’s a cost for using it in the first place. In the
>> case
>> of prefast, it’s essentially unreadable code, which gets into
>> cultural/religious coding issues, which are also real and important.
>>
>>
>>
>> mm
>>
>>
>> -----Original Message-----
>> From: xxxxx@lists.osr.com
>> [mailto:xxxxx@lists.osr.com] On Behalf Of Volodymyr M.
>> Shcherbyna
>> Sent: Wednesday, October 27, 2010 6:18 PM
>> To: Windows System Software Devs Interest List
>> Subject: [ntdev] Quality of kernel drivers of well-known vendors like
>> VMWare, McAfee, Juniper. Why does it suck so much?
>>
>> Hello everyone,
>>
>> Just want to share some feedback I have regarding third party drivers I
>> met
>> sometimes during my tests. Interesting to know, is it only happening to
>> me
>
>> ?
>>
>> Or you also saw something simular when dealing with them?
>>
>> Before releasing my code I always do heavy stress testing with
>> antiviruses/fws and I set verifier to verify mine and av/fw driver with
>> maximum checks. Mine driver is a typical TDI filter with some additional
>> functionality (I set PsLoadImageNotify callbacks to track process
>> creation
>> events). (please, no need to point that TDI is depricated, I know that, I
>> am
>>
>> already taking care of it. This is not the purpose of this post)
>>
>> So, testing with firewalls I always get BSODs with McAfee, Trend Micro
>> (with
>>
>> Trend Micro I have less problems, though) in their TDI filters. Even if I
>> do
>>
>> tests on clean machine (i.e., clean install of XP or Vista and there is
>> NO
>> my driver) McAfee BSOD’s with pool corruption assert within 10 minutes of
>> stress testing.
>>
>> (I was also troubleshooting issues with Juniper VPN client which is btw a
>> TDI filter. You enable verifier, run it, and you got immidiate BSOD. IIRC
>> it
>>
>> was related to not marking IRP pending, or something simular.)
>>
>> Mine stress test tools are simple: create a lot of processes and make a
>> lot
>> of connections, and it just dies :). I’ve sent request to McAfee about
>> this
>> problem, but they never actually resolved it. But this is not the point,
>> I
>> am interested to know your opinion about quality of those guys.
>>
>> Today I was experimenting with my VmWare Workstation at home in Windows 7
>> x64 and I got a BSOD when doing verification of vmx86.sys driver. The
>> test
>> case is very simple:
>>
>> 1. Fire verifier and set maximum checks for vmx86.sys
>>
>> 2. Reboot
>>
>> 3. Start any x86 virtual machine, and voila, you have the following:
>>
>> DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
>> A device driver attempting to corrupt the system has been caught. This
>> is
>> because the driver was specified in the registry as being suspect (by the
>> administrator) and the kernel has enabled substantial checking of this
>> driver.
>> If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and
>> 0xA
>> will
>> be among the most commonly seen crashes.
>> Arguments:
>> Arg1: 00000000000000f6, Referencing user handle as KernelMode.
>> Arg2: 00000000000002e0, Handle value being referenced.
>> Arg3: fffffa8005b9fb30, Address of the current process.
>> Arg4: fffff880062da398, Address inside the driver that is performing the
>> incorrect reference.
>>
>> Debugging Details:
>> ------------------
>>
>>
>> BUGCHECK_STR: 0xc4_f6
>>
>> CUSTOMER_CRASH_COUNT: 1
>>
>> DEFAULT_BUCKET_ID: VERIFIER_ENABLED_VISTA_MINIDUMP
>>
>> PROCESS_NAME: vmware-vmx.exe
>>
>> CURRENT_IRQL: 0
>>
>> LAST_CONTROL_TRANSFER: from fffff800035493dc to fffff800030bf740
>>
>> STACK_TEXT:
>> fffff88007953f48 fffff800035493dc : 00000000000000c4 00000000000000f6
>> 00000000000002e0 fffffa8005b9fb30 : nt!KeBugCheckEx
>> fffff88007953f50 fffff8000355eae4 : 00000000000002e0 fffffa8005b9fb30
>> 0000000000000007 fffff78000001000 :
>> nt!VerifierBugCheckIfAppropriate+0x3c
>> fffff88007953f90 fffff8000331ab40 : fffff6fc00019ae8 fffff880079541f0
>> fffffa8004619500 fffffa800c0a2cc8 : nt!VfCheckUserHandle+0x1b4
>> fffff88007954070 fffff800033b8ab5 : 0000000000000000 0000000000000000
>> 0000000000000000 0000007fffffff00 : nt! ?? ::NNGAKEGL::string'+0x20e2e >> fffff88007954140 fffff800033bde4d : fffffa800c0a2b10 fffff880079542a0 >> fffff6fb00000040 fffffa8004619510 : nt!ObpLookupObjectName+0x1b5 >> fffff88007954240 fffff8000335d654 : fffff880079544c8 0000000000000000 >> fffff88007954300 fffff800030bd93d : nt!ObOpenObjectByName+0x1cd >> fffff880079542f0 fffff8000335d72e : fffff88007954760 fffff880000f003f >> fffff88007954780 fffff8000335d700 : nt!CmCreateKey+0x2e1 >> fffff88007954460 fffff800030be993 : 0000000000000000 0000000000000000 >> 0000000000000000 0000000000000000 : nt!NtCreateKey+0x2e >> fffff880079544b0 fffff800030baf30 : fffff8000354fb49 fffff88007954780 >> fffff880079548a0 0000000000261b00 : nt!KiSystemServiceCopyEnd+0x13 >> fffff880079546b8 fffff8000354fb49 : fffff88007954780 fffff880079548a0 >> 0000000000261b00 fffff880062e5590 : nt!KiServiceLinkage >> fffff880079546c0 fffff880062da398 : 0000000000000000 fffff880062e5590 >> 0000000000261b00 fffff880079548a0 : nt!VfZwCreateKey+0x99 >> fffff88007954720 0000000000000000 : fffff880062e5590 0000000000261b00 >> fffff880079548a0 0000000000000000 : vmx86+0x3398 >> >> >> STACK_COMMAND: kb >> >> FOLLOWUP_IP: >> vmx86+3398 >> fffff880062da398 413bc4 cmp eax,r12d
>>
>> SYMBOL_STACK_INDEX: b
>>
>> SYMBOL_NAME: vmx86+3398
>>
>> FOLLOWUP_NAME: MachineOwner
>>
>> MODULE_NAME: vmx86
>>
>> IMAGE_NAME: vmx86.sys
>>
>> DEBUG_FLR_IMAGE_TIMESTAMP: 4a85fa8c
>>
>> FAILURE_BUCKET_ID: X64_0xc4_f6_VRF_vmx86+3398
>>
>> BUCKET_ID: X64_0xc4_f6_VRF_vmx86+3398
>>
>> Followup: MachineOwner
>> ---------
>>
>> This is not the first bug check I have, I also had something related to
>> memory corruption.
>>
>> Is it a fault of verifier? I doubt. On the other hand, these companies
>> (McAfee, VmWare, Juniper) earn millions and spend millions on R&D teams,
>> how
>>
>> is it possible that they just never do Driver Verifier tests? (At least)
>> …
>>
>> Hope to hear some feedback on this : )
>>
>> –
>> Volodymyr
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>>
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>

john_bloe · October 27, 2010, 7:56pm

0xc4-0xf6 bugchecks are perhaps not high enough in fix list. You can read more about this bugcheck @http://winprogger.com/?p=651.

Satya
http://www.winprogger.com

Rossetoecioccolato · October 27, 2010, 8:00pm

Yes. The driver verifier is a useful tool for rootkit authors as well for
the same reason. If I were designing malware (which I am not), I would be
running these third party drivers with the verifier on to learn how to
exploit them. Thanks for the link.

Regards,

George.

wrote in message news:xxxxx@ntdev…
> 0xc4-0xf6 bugchecks are perhaps not high enough in fix list. You can read
> more about this bugcheck @http://winprogger.com/?p=651.
>
> Satya
> http://www.winprogger.com
>

mm1 · October 27, 2010, 8:04pm

Out of context, I agree with what you’re saying.

In the context of a stable, VERY useful, HUGELY successful product, built
using particular procedures, developed in a particular culture for years,
and limited resources and competing wants, I don’t agree with what you’re
saying necessarily. The time one spends tracking down any particular
problem means that something else doesn’t get done, and in the end, there
most definitely is such a thing as working well enough. There has to be,
unless you’ve got unlimited resources.

Also, why stop with Driver Verifier? Why not include lint, sdv, bullseye,
rational, et. c.? Surely they will find something wrong? You’re either
assuming that all of this is either obviously PROFITABLE or free.

So, what I was saying was that in my opinion, in the case of VMWare
Workstation, perhaps that’s what’s going on. I have no idea, but I do know
that I would never extend this statement to include the av’s mentioned, for
example.

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of George M. Garner Jr.
Sent: Wednesday, October 27, 2010 7:44 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Quality of kernel drivers of well-known vendors like
VMWare, McAfee, Juniper. Why does it suck so much?

Martin,

I am saying:

When the driver verifier bugchecks there is usually a good reason for it;

and

The most insidious bugs have no apparent effects until one day random
data corruption hits the right spot and the computer orders a drone to fire
a missile… And then you end up the lead story on the nightly news.

The fact that some buggy driver appears to “work” is no indication of what
might happen down the road.

The driver verifier can be a pain in the ass at times. However, a pain in
the ass during development is better than a pain in the ass down the road.
Just my 2 centimes worth.

Regards,

George.

“Martin O’Brien” wrote in message
news:xxxxx@ntdev…
> You’re saying that it doesn’t matter if the product works or not, unless
> they run Driver Verifier, it’s crap?
>
>
> mm
>
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of George M. Garner
> Jr.
> Sent: Wednesday, October 27, 2010 7:22 PM
> To: Windows System Software Devs Interest List
> Subject: Re:[ntdev] Quality of kernel drivers of well-known vendors like
> VMWare, McAfee, Juniper. Why does it suck so much?
>
> Bull-$@#! The driver verifier is a part of reasonable due care. And I am
> personally thankful for it. It has saved me a lot of headaches down the
> road. We develop with the verifier (and Prefast) ON.
>
> Regards,
>
> George.
>
> “Martin O’Brien” wrote in message
> news:xxxxx@ntdev…
>> >On the other hand, these companies (McAfee, VmWare, Juniper) earn
>> >millions
>> and spend millions on R&D teams, how is it possible that they just never
>> do
>> Driver Verifier tests? (At least) …
>>
>> Although Driver Verifier has been around since W2K (I think), it
>> certainly
>> hasn’t been in mainstream usage anywhere near that long, in my opinion,
>> and
>> if you’re considering PREFast as well, that’s far newer and in my
>> experience
>> unfortunately not at all commonly used.
>>
>> While I hear what you’re saying in general, I think that it’s important
>> to
>> observe that there are many ways to go about debugging, and stable
>> products
>> have definitely been built with the use of either Driver Verifier and/or
>> PREFast.
>>
>> For example, I use VMWare workstation every day, and while it certainly
>> has
>> some issues like all software, I would call it a stable, at least for my
>> purposes. I’m not saying that that error you mentioned below isn’t a
>> problem (I have no idea), but in the presence of limited resources to
>> throw
>> at all problems, making code compliant with tools after the fact
>> (hypothetically) can be hard to justify (and is just not possible in some
>> cases) if everything is in working order in practice and you have other
>> fish
>> to fry on top of it.
>>
>> This is in my opinion far less of a problem with verifier than it is with
>> PREFast, where it can be painfully acute how large the cost of getting
>> old
>> code to be prefast clean can be (or lint, et. c.), not to mention that as
>> with any tools, there’s a cost for using it in the first place. In the
>> case
>> of prefast, it’s essentially unreadable code, which gets into
>> cultural/religious coding issues, which are also real and important.
>>
>>
>>
>> mm
>>
>>
>> -----Original Message-----
>> From: xxxxx@lists.osr.com
>> [mailto:xxxxx@lists.osr.com] On Behalf Of Volodymyr M.
>> Shcherbyna
>> Sent: Wednesday, October 27, 2010 6:18 PM
>> To: Windows System Software Devs Interest List
>> Subject: [ntdev] Quality of kernel drivers of well-known vendors like
>> VMWare, McAfee, Juniper. Why does it suck so much?
>>
>> Hello everyone,
>>
>> Just want to share some feedback I have regarding third party drivers I
>> met
>> sometimes during my tests. Interesting to know, is it only happening to
>> me
>
>> ?
>>
>> Or you also saw something simular when dealing with them?
>>
>> Before releasing my code I always do heavy stress testing with
>> antiviruses/fws and I set verifier to verify mine and av/fw driver with
>> maximum checks. Mine driver is a typical TDI filter with some additional
>> functionality (I set PsLoadImageNotify callbacks to track process
>> creation
>> events). (please, no need to point that TDI is depricated, I know that, I
>> am
>>
>> already taking care of it. This is not the purpose of this post)
>>
>> So, testing with firewalls I always get BSODs with McAfee, Trend Micro
>> (with
>>
>> Trend Micro I have less problems, though) in their TDI filters. Even if I
>> do
>>
>> tests on clean machine (i.e., clean install of XP or Vista and there is
>> NO
>> my driver) McAfee BSOD’s with pool corruption assert within 10 minutes of
>> stress testing.
>>
>> (I was also troubleshooting issues with Juniper VPN client which is btw a
>> TDI filter. You enable verifier, run it, and you got immidiate BSOD. IIRC
>> it
>>
>> was related to not marking IRP pending, or something simular.)
>>
>> Mine stress test tools are simple: create a lot of processes and make a
>> lot
>> of connections, and it just dies :). I’ve sent request to McAfee about
>> this
>> problem, but they never actually resolved it. But this is not the point,
>> I
>> am interested to know your opinion about quality of those guys.
>>
>> Today I was experimenting with my VmWare Workstation at home in Windows 7
>> x64 and I got a BSOD when doing verification of vmx86.sys driver. The
>> test
>> case is very simple:
>>
>> 1. Fire verifier and set maximum checks for vmx86.sys
>>
>> 2. Reboot
>>
>> 3. Start any x86 virtual machine, and voila, you have the following:
>>
>> DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
>> A device driver attempting to corrupt the system has been caught. This
>> is
>> because the driver was specified in the registry as being suspect (by the
>> administrator) and the kernel has enabled substantial checking of this
>> driver.
>> If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and
>> 0xA
>> will
>> be among the most commonly seen crashes.
>> Arguments:
>> Arg1: 00000000000000f6, Referencing user handle as KernelMode.
>> Arg2: 00000000000002e0, Handle value being referenced.
>> Arg3: fffffa8005b9fb30, Address of the current process.
>> Arg4: fffff880062da398, Address inside the driver that is performing the
>> incorrect reference.
>>
>> Debugging Details:
>> ------------------
>>
>>
>> BUGCHECK_STR: 0xc4_f6
>>
>> CUSTOMER_CRASH_COUNT: 1
>>
>> DEFAULT_BUCKET_ID: VERIFIER_ENABLED_VISTA_MINIDUMP
>>
>> PROCESS_NAME: vmware-vmx.exe
>>
>> CURRENT_IRQL: 0
>>
>> LAST_CONTROL_TRANSFER: from fffff800035493dc to fffff800030bf740
>>
>> STACK_TEXT:
>> fffff88007953f48 fffff800035493dc : 00000000000000c4 00000000000000f6
>> 00000000000002e0 fffffa8005b9fb30 : nt!KeBugCheckEx
>> fffff88007953f50 fffff8000355eae4 : 00000000000002e0 fffffa8005b9fb30
>> 0000000000000007 fffff78000001000 :
>> nt!VerifierBugCheckIfAppropriate+0x3c
>> fffff88007953f90 fffff8000331ab40 : fffff6fc00019ae8 fffff880079541f0
>> fffffa8004619500 fffffa800c0a2cc8 : nt!VfCheckUserHandle+0x1b4
>> fffff88007954070 fffff800033b8ab5 : 0000000000000000 0000000000000000
>> 0000000000000000 0000007fffffff00 : nt! ?? ::NNGAKEGL::string'+0x20e2e >> fffff88007954140 fffff800033bde4d : fffffa800c0a2b10 fffff880079542a0 >> fffff6fb00000040 fffffa8004619510 : nt!ObpLookupObjectName+0x1b5 >> fffff88007954240 fffff8000335d654 : fffff880079544c8 0000000000000000 >> fffff88007954300 fffff800030bd93d : nt!ObOpenObjectByName+0x1cd >> fffff880079542f0 fffff8000335d72e : fffff88007954760 fffff880000f003f >> fffff88007954780 fffff8000335d700 : nt!CmCreateKey+0x2e1 >> fffff88007954460 fffff800030be993 : 0000000000000000 0000000000000000 >> 0000000000000000 0000000000000000 : nt!NtCreateKey+0x2e >> fffff880079544b0 fffff800030baf30 : fffff8000354fb49 fffff88007954780 >> fffff880079548a0 0000000000261b00 : nt!KiSystemServiceCopyEnd+0x13 >> fffff880079546b8 fffff8000354fb49 : fffff88007954780 fffff880079548a0 >> 0000000000261b00 fffff880062e5590 : nt!KiServiceLinkage >> fffff880079546c0 fffff880062da398 : 0000000000000000 fffff880062e5590 >> 0000000000261b00 fffff880079548a0 : nt!VfZwCreateKey+0x99 >> fffff88007954720 0000000000000000 : fffff880062e5590 0000000000261b00 >> fffff880079548a0 0000000000000000 : vmx86+0x3398 >> >> >> STACK_COMMAND: kb >> >> FOLLOWUP_IP: >> vmx86+3398 >> fffff880062da398 413bc4 cmp eax,r12d
>>
>> SYMBOL_STACK_INDEX: b
>>
>> SYMBOL_NAME: vmx86+3398
>>
>> FOLLOWUP_NAME: MachineOwner
>>
>> MODULE_NAME: vmx86
>>
>> IMAGE_NAME: vmx86.sys
>>
>> DEBUG_FLR_IMAGE_TIMESTAMP: 4a85fa8c
>>
>> FAILURE_BUCKET_ID: X64_0xc4_f6_VRF_vmx86+3398
>>
>> BUCKET_ID: X64_0xc4_f6_VRF_vmx86+3398
>>
>> Followup: MachineOwner
>> ---------
>>
>> This is not the first bug check I have, I also had something related to
>> memory corruption.
>>
>> Is it a fault of verifier? I doubt. On the other hand, these companies
>> (McAfee, VmWare, Juniper) earn millions and spend millions on R&D teams,
>> how
>>
>> is it possible that they just never do Driver Verifier tests? (At least)
>> …
>>
>> Hope to hear some feedback on this : )
>>
>> –
>> Volodymyr
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>>
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Rossetoecioccolato · October 27, 2010, 8:15pm

Martin,

A lot of AV’s are basically rootkits. So I would include especially the
AV’s. A few years back I could tell you which rootkit you had on your
computer by which AV you were running. Different rootkits were adapted to,
even required, different AV’s. Nowadays, rootkits provide “support” for
most common AV’s.

Regards,

George.

mm1 · October 27, 2010, 8:17pm

You lost me - what does this have to do with vmware (allegedly) not running
under driver verifier?

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of George M. Garner Jr.
Sent: Wednesday, October 27, 2010 8:15 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Quality of kernel drivers of well-known vendors like
VMWare, McAfee, Juniper. Why does it suck so much?

Martin,

A lot of AV’s are basically rootkits. So I would include especially the
AV’s. A few years back I could tell you which rootkit you had on your
computer by which AV you were running. Different rootkits were adapted to,
even required, different AV’s. Nowadays, rootkits provide “support” for
most common AV’s.

Regards,

George.

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Rossetoecioccolato · October 27, 2010, 8:31pm

Martin,

I have no idea, but I do know that I would never extend this statement
to include the av’s mentioned, for example. <

I was commenting on this line. However, after re-reading it I am not sure
whether you are saying that the AV’s don’t need to be verified or that they
do.

Sorry if I misunderstood you.

Regards,

George.

mm1 · October 27, 2010, 8:34pm

I was saying that I wouldn’t extend my comments about vmware workstation
being stable and so forth to most of (if any) av’s.

mm
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of George M. Garner Jr.
Sent: Wednesday, October 27, 2010 8:31 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Quality of kernel drivers of well-known vendors like
VMWare, McAfee, Juniper. Why does it suck so much?

Martin,

I have no idea, but I do know that I would never extend this statement
to include the av’s mentioned, for example. <

I was commenting on this line. However, after re-reading it I am not sure
whether you are saying that the AV’s don’t need to be verified or that they
do.

Sorry if I misunderstood you.

Regards,

George.

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Pavel_A1 · October 27, 2010, 8:52pm

The verifier found bugs on win7 can be at least partially explained
by improvement of the verifier itself.
It seems to check more (or better) than in Vista and XP.

/me remembers testing a certain driver before the past filter plugfest; it passed
clean on XP and Vista sp1, but showed lots of bugs on win7 (beta then).
The “Referencing user handle as KernelMode” was among the newly
found bugs.
/* we managed to fix all by the plugfest, and Alex C. said he was surprised */

So maybe they just have not tested on win7 enough.
The vmware bug is really sad; /me used to believe they are rock solid…
–pa

Rossetoecioccolato · October 27, 2010, 8:56pm

Martin,

Sorry. My mistake. My sincerest apologies.

Regards,

George.

mm1 · October 27, 2010, 8:57pm

No worries.

Sorry if I was short as well.

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of George M. Garner Jr.
Sent: Wednesday, October 27, 2010 8:56 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Quality of kernel drivers of well-known vendors like
VMWare, McAfee, Juniper. Why does it suck so much?

Martin,

Sorry. My mistake. My sincerest apologies.

Regards,

George.

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Rossetoecioccolato · October 27, 2010, 9:01pm

I give MS a lot of credit for this too. Hopefully they will move some of
these bugchecks into the OS in a future release, because, as we see, driver
writers ignore secure programming guidelines; and the rest of us pay the
price.

Regards,

George.

wrote in message news:xxxxx@ntdev…
> The verifier found bugs on win7 can be at least partially explained
> by improvement of the verifier itself.
> It seems to check more (or better) than in Vista and XP.
>