Verifier causes the OS to not work properly?

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2051

No, this is not “Verifier causes the OS to not work properly?” but
“VMware causes the OS to not work properly?”. This is an example of why
some of us, don’t use virtual machines for most testing, since over the
years I have seen too many surprises caused by the hypervisor.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

xxxxx@terabyteunlimited.com” wrote in message
news:xxxxx@ntdev:

> http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2051

I totally agree Don, my opinion of VMWare just took a step down.

Does this mean you can NEVER run driver verifier on Windows 7 under VMWare?

Last I knew, Hyper-V guests would perfectly run with driver verifier and would run the checked OS.

Jan

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Tuesday, March 13, 2012 1:16 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Verifier causes the OS to not work properly?

No, this is not “Verifier causes the OS to not work properly?” but “VMware causes the OS to not work properly?”. This is an example of why some of us, don’t use virtual machines for most testing, since over the years I have seen too many surprises caused by the hypervisor.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

xxxxx@terabyteunlimited.com” wrote in message
news:xxxxx@ntdev:

> http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&c
> md=displayKC&externalId=2051


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Jan,

At least one version of Xen I know about handled Windows verifier
fine also. VmWare is what turned me off on using a hypervisor for
debugging, I have had multiple instances of things screwing up because
of VmWare.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

“Jan Bottorff” wrote in message
news:xxxxx@ntdev:

> I totally agree Don, my opinion of VMWare just took a step down.
>
> Does this mean you can NEVER run driver verifier on Windows 7 under VMWare?
>
> Last I knew, Hyper-V guests would perfectly run with driver verifier and would run the checked OS.
>
> Jan
>
> -----Original Message-----
> From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
> Sent: Tuesday, March 13, 2012 1:16 PM
> To: Windows System Software Devs Interest List
> Subject: Re:[ntdev] Verifier causes the OS to not work properly?
>
> No, this is not “Verifier causes the OS to not work properly?” but “VMware causes the OS to not work properly?”. This is an example of why some of us, don’t use virtual machines for most testing, since over the years I have seen too many surprises caused by the hypervisor.
>
>
> Don Burn
> Windows Filesystem and Driver Consulting
> Website: http://www.windrvr.com
> Blog: http://msmvps.com/blogs/WinDrvr
>
>
>
> “xxxxx@terabyteunlimited.com” wrote in message
> news:xxxxx@ntdev:
>
> > http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&c
> > md=displayKC&externalId=2051
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

… it was actually the host that got the BSOD when it was running verifier, not the guests … yikes.

So I’m confused now, you’re saying if you run driver verifier in a Windows 7 vm running under (some unknown version) of a VMware hypervisor, then the hypervisor crashes?

Or are you saying if you run VMWare desktop as an app under Windows 7, and enable driver verifier on Windows 7, the system crashes.

The VMware bug pages says products(s) VMWare Player, VMWare Server, VMWare Workstation. Some looking on the VMWare site for what “VMWare Server” actually is lists it as an end-of-life product. VMWare Workstation 8, looks like a current product. I would have to interpret this issue in the context of the second paragraph above, not the current standalone server hypervisor product.

Jan

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@terabyteunlimited.com
Sent: Tuesday, March 13, 2012 5:54 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Verifier causes the OS to not work properly?

… it was actually the host that got the BSOD when it was running verifier, not the guests … yikes.


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

> … it was actually the host that got the BSOD when it was running verifier, not the guests … yikes.

…which simply means there is some bug in VMWare’s kernel component so that it does not work under Verifier…

Anton Bassov

> VmWare is what turned me off on using a hypervisor for debugging, I have had multiple instances

of things screwing up because of VmWare.

Probably, because your code was not perfect. I DO know that VMWare does not always respond the way a physical machine does, but normally it happens when you try to do something “not-so-standard”
(i.e.access the hardware not the way it is supposed to be accessed by the software). For example, very long time ago I was simulating a keystroke at the level of i8042prt controller. What I was doing was raising interrupt via local APIC’s ICR and filling the buffer in interrupt handler stub, i.e. before ISR gets invoked. Under the physical machine everything worked fine, but under VMWare the buffer was always empty so that ISR subsequently dismissed interrupt as a spurious one…

Anton Bassov

On 13/03/2012 23:22, Jan Bottorff wrote:

I totally agree Don, my opinion of VMWare just took a step down.

Does this mean you can NEVER run driver verifier on Windows 7 under VMWare?

Last I knew, Hyper-V guests would perfectly run with driver verifier and would run the checked OS.

The problem is with enabling the verifier on the host, not a guest.


Bruce Cran

Anton,

I never started out to use it. Mainly I got drivers that others
wrote that were “fully tested” but it turned out to be under VmWare, of
course this meant that they really weren’t tested at all!. Of course
along the way, I saw a few cases where the host wiped out the hypervisor
also. Bottom line, is I never found I could trust a VmWare environment.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

xxxxx@hotmail.com” wrote in message
news:xxxxx@ntdev:

> > VmWare is what turned me off on using a hypervisor for debugging, I have had multiple instances
> > of things screwing up because of VmWare.
>
> Probably, because your code was not perfect. I DO know that VMWare does not always respond the way a physical machine does, but normally it happens when you try to do something “not-so-standard”
> (i.e.access the hardware not the way it is supposed to be accessed by the software). For example, very long time ago I was simulating a keystroke at the level of i8042prt controller. What I was doing was raising interrupt via local APIC’s ICR and filling the buffer in interrupt handler stub, i.e. before ISR gets invoked. Under the physical machine everything worked fine, but under VMWare the buffer was always empty so that ISR subsequently dismissed interrupt as a spurious one…
>
>
> Anton Bassov

> Mainly I got drivers that others wrote that were “fully tested” but it turned out to be

under VmWare, of course this meant that they really weren’t tested at all!

Well, it depends on driver type. Certainly, as long as we are speaking about the hardware devices, testing a driver under VMWAre is exactly the same thing as not testing it at all. However, I just cannot imagine how someone can possibly develop a driver for a hardware device under a VM of any description, unless he/she is able to extend it with emulation of the target device( which obviously cannot be done under a proprietary product like VMWare). For those drivers that don’t touch the hardware VMWAre works just fine, at least in my experience, and I think that most drivers that are tested under VMWAre fall into this category…

along the way, I saw a few cases where the host wiped out the hypervisor also.

Could you please expand it a bit…

Bottom line, is I never found I could trust a VmWare environment.

Well, you don’t really trust anything that comes from any source other than MSFT, do you…

Anton Bassov

Anton,

The drivers were software only. In the past I have encountered:

  1. Multi-processing that is not really multi-processing. I.E. even
    though VmWare gives you multiple processors in a hosted environment it
    impacts scheduling enough that essentially testing of locking and
    synchronization was worthless.

  2. Somewhat strange behavior in memory handling. At least enough that
    you could get into trouble.

  3. Timing issues (not synchronization).

This does not count the cases where valid Windows drivers in the hosted
environment have crashed VmWare.

Bottom line is I tell my clients if they are getting a software only
driver and it has primarily been tested in a VmWare environment, ASSUME
THAT THE DRIVER HAS NOT BEEN TESTED AT ALL.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

xxxxx@hotmail.com” wrote in message
news:xxxxx@ntdev:

> > Mainly I got drivers that others wrote that were “fully tested” but it turned out to be
> > under VmWare, of course this meant that they really weren’t tested at all!
>
> Well, it depends on driver type. Certainly, as long as we are speaking about the hardware devices, testing a driver under VMWAre is exactly the same thing as not testing it at all. However, I just cannot imagine how someone can possibly develop a driver for a hardware device under a VM of any description, unless he/she is able to extend it with emulation of the target device( which obviously cannot be done under a proprietary product like VMWare). For those drivers that don’t touch the hardware VMWAre works just fine, at least in my experience, and I think that most drivers that are tested under VMWAre fall into this category…
>
> > along the way, I saw a few cases where the host wiped out the hypervisor also.
>
> Could you please expand it a bit…
>
>
> > Bottom line, is I never found I could trust a VmWare environment.
>
> Well, you don’t really trust anything that comes from any source other than MSFT, do you…
>
>
> Anton Bassov

I like the way that they describe the problem too - I wonder if Microsoft
would agree that the memory check erroneously finds a problem

wrote in message news:xxxxx@ntdev…

… it was actually the host that got the BSOD when it was running verifier,
not the guests … yikes.

> Multi-processing that is not really multi-processing. I.E. even though VmWare gives you multiple

processors in a hosted environment it impacts scheduling enough that essentially testing of locking
and synchronization was worthless.

Unfortunately, this problem seems to be inevitable under ANY hypervisor. In order to solve it hypervisor has to ensure that all VCPUs of a given guest run simultaneously all the time, which is obviously going to be terribly inefficient, although feasible, under Type 1 hypervisor, and simply infeasible task under Type 2 one
( i.e. something VMWare workstation happens to be), because in order to achieve it hypervisor has to be able to control host’s scheduler…

However, as Joe has recently pointed out, when it comes to locking and synchronization testing is worthless exercise in itself, because it can only detect the presence of bugs but never proves absence of any - everything that deals with this issues has to be solved only via thorough code review…

  1. Timing issues (not synchronization).

Actually, I don’t even understand what you are trying to say - we are not speaking about RTOSes here, are we. Therefore, we should not expect to get any meaningful timing results. If you mean things like QPC…
well, depending on HAL, you can get skewed results on the physical machine as well, so that you should not use its results as a measure of elapsed time and reserve it to the things like timestamps…

Anton Bassov

Did I READ that right?

Testing is WORTHLESS because it can *only* DETECT THE PRESENCE OF BUGS?

I don’t know about you, but that’s why I test my code: To find bugs.

Try as I might, I can never envision all the conditions and interactions under which my code will run. Testing often reveals scenarios I hadn’t considered, or hadn’t sufficiently or correctly considered.

Given the complexity and ever-changing nature of the interfaces with which we interact, NO practical testing one could envision could EVER prove the absence of bugs in a driver. None.

W. Edwards Deming aside, I’ve never heard of somebody saying “nah, no point in testing my code. It won’t prove my code is bug free, so… you know… I’ll just do a code review and call it good.”

Well, now that I think of it: I did have an engineer work for me once who actually thought he could find ALL his bugs via personal code inspection. Never added a DbgPrint or any tracing to his code, because, he asserted that he’d never need it. Just a desk check followed by a simple walk-through and his (very complex) component would be sure to work correctly.

I fired him.

WTF are you talking about Anton?

Peter
OSR

Well don’t do that. Why would you enable verifer on your development
system or on the host system for test VMs?

Verifier bugchecks when it finds a *potential* problem, it is only
suitable for use on test systems that can tolerate crashes.

“Every time I hit myself with a hammer it hurts”.

Mark Roddy

On Wed, Mar 14, 2012 at 3:50 AM, Bruce Cran wrote:
> On 13/03/2012 23:22, Jan Bottorff wrote:
>>
>> I totally agree Don, my opinion of VMWare just took a step down.
>>
>> Does this mean you can NEVER run driver verifier on Windows 7 ?under
>> VMWare?
>>
>> Last I knew, Hyper-V guests would perfectly run with driver verifier and
>> would run the checked OS.
>
>
> The problem is with enabling the verifier on the host, not a guest.
>
> –
> Bruce Cran
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

Peter,

WTF are you talking about Anton?

I am talking about synchronization bugs. Unlike other bugs like memory corruption, buffer overruns, null pointer dereference, etc these bugs are particularly nasty because they may reveal themselves, without exaggeration, once per billions or even trillions of runs. For example, check http://www.osronline.com/showThread.cfm?link=218555. I suggested a locking algorithm, and someone suggested the “improvement”
to it. How long do you think would it take to discover such a bug in a testing environment (assuming interrupts are masked it would require SMI firing exactly during the period that is measured in sub-nanosecond range)???

This is what I am speaking about…

This is what I call blowing the things out of proportion, which normally results in ridiculous things…

Anton Bassov

Three days? Two weeks? A long time?

The point is, you don’t have to do live testing OR do desk checking. Since perestroika we can now do both!! Hooray!

To say that “when it comes to locking and synchronization testing is worthless exercise” is just plain silly. To say this is so because “[testing] … never proves absence of any [bugs]” – that testing can’t prove a negative – is even more silly.

Desk checking good. Testing good. Desk checking PLUS testing… BETTER! Desk checking plus testing plus static analysis… EVEN BETTER YET.

Peter
OSR

“are you saying if you run VMWare desktop as an app under Windows 7, and enable driver verifier on Windows 7, the system crashes.”

It’s not my system, I don’t use that software as I find it too intrusive and noticeable, this was on another system for someone working for me. but yes, they have VMWare installed on Win7 x64, can’t enable verifier and start a guest or it crashes the host (I believe that’s only when).

Do you have a kernel debugger attached to the computer that is crashing?
Do you need to run driver verifier on the VM host? Assuming that you have DV enabled for all drivers, could you scope it to a smaller set of drivers?

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@terabyteunlimited.com
Sent: Wednesday, March 14, 2012 11:12 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Verifier causes the OS to not work properly?

“are you saying if you run VMWare desktop as an app under Windows 7, and enable driver verifier on Windows 7, the system crashes.”

It’s not my system, I don’t use that software as I find it too intrusive and noticeable, this was on another system for someone working for me. but yes, they have VMWare installed on Win7 x64, can’t enable verifier and start a guest or it crashes the host (I believe that’s only when).


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer