Comments

• Tim_Roberts Member - All Emails Posts: 14,563

  July 2021

  Do you have a driver in this game? Is VITLService one of your services? The bugcheck seems pretty clear; a driver returned without lowering the IRQL. Either a dangling spin lock, or some foolish manual IRQL manipultion.

  Tim Roberts, [email protected]
  Providenza & Boekelheide, Inc.

  0 · Share on Facebook
• Erik_Dabrowsky Member - All Emails Posts: 37

  July 2021

  Yes, there are two drivers involved and one is ours. Ours is a legacy WDM software only driver that interacts with another vendor's driver via function pointers exchanged during initialization (driver entry). The vendor's driver interacts with their hardware and ours uses their API.

  So far I haven't been able to reproduce it in our environment, but I'm able to reproduce it consistently in a customer's environment.
  (Don't you love those cases?)

  At this point I'm not sure if our driver is causing the crash or theirs. I was hoping the crash dump would shed some light on that. In the past when I found a bug in our code it was pretty obvious by the stack text in the crash dump.

  I've checked to make sure KeReleaseSpinLock() is called before IoCompleteRequest() in any cases where the spinlock is acquired.

  Would this happen if the vendor's driver is calling one of our functions at something higher than DISPATCH_LEVEL and then we call IoCompleteRequest() or would that be a different bug check error code? I can't imagine they would do that because that would mean they would be calling our function from their ISR.

  Thanks for your input on this.
  Erik

  0 · Share on Facebook
• Tim_Roberts Member - All Emails Posts: 14,563

  July 2021

  Do you have enough logging to tell which request triggers this? That might help. IoCompleteRequest can be called at dispatch level just fine. This is not about completion, this is happening when a driver returns back to the user.

  Tim Roberts, [email protected]
  Providenza & Boekelheide, Inc.

  0 · Share on Facebook
• Erik_Dabrowsky Member - All Emails Posts: 37

  July 2021 edited July 2021

  So far still no luck reproducing this in our environment.
  I'm going to see if the customer will be gracious enough to let me try and debug it in their environment.

  I've been looking at the cancellation routine code, thinking maybe the user mode app is crashing with an IRP in flight.

  I see something that looks a little odd in the cancel routine:

  OldIRQL = KeGetCurrentIrql();

  if (OldIRQL < DISPATCH_LEVEL)
  KeAcquireSpinLock(&DeviceExtension->TSPISpinLock, &OldIRQL);
  else
  KeAcquireSpinLockAtDpcLevel(&DeviceExtension->TSPISpinLock);

  ... (do some stuff with spinlock held)

  if (OldIRQL < DISPATCH_LEVEL)
  KeReleaseSpinLock(&DeviceExtension->TSPISpinLock, OldIRQL);
  else
  KeReleaseSpinLockFromDpcLevel(&DeviceExtension->TSPISpinLock);

  I don't see the advantage of using KeAcquireSpinLockAtDpcLevel/KeReleaseSpinLockFromDpcLevel in a cancellation routine.

  0 · Share on Facebook
• Erik_Dabrowsky Member - All Emails Posts: 37

  July 2021

  So I was finally able to reproduce this in our environment and I found that downgrading the vendor's driver to an older version stopped it from crashing. I'm still not sure if their driver is crashing or if it's behaving in a way that causes ours to crash. Will be debugging it further now that I can reproduce it.

  0 · Share on Facebook
• Erik_Dabrowsky Member - All Emails Posts: 37

  July 2021

  Just thought I would follow up with the resolution to this issue.

  There was a certain condition in our code that would cause KeReleaseSpinLock to get called twice on the same lock.
  I was surprised that SDV didn't flag it.

  The way I finally found it was by writing wrapper routines around KeAcquireSpinLock and KeReleaseSpinLock that incremented and decremented a counter. If the counter ever went above 1 or below 0, I manually bugchecked the system.

  This was tricky to debug because there wasn't an immediate crash and in our case the timing had to be just right for it to travel down the code path where the 2nd release of the lock occurred.

  0 · Share on Facebook
• Peter_Viscarola_(OSR) Administrator Posts: 9,077

  July 2021

  Thanks for following-up. That's always helpful for the archives.

  KeAcquireSpinLock and KeReleaseSpinLock are really very simple functions, and -- as you've discovered -- do no error checking. It IS disappointing that SDV didn't catch this. Hmmmm...

  Again, thanks for following up. Great bug!

  Peter

  Peter Viscarola
  OSR
  @OSRDrivers

  0 · Share on Facebook