Class compliant MIDI / vendor specific device not removed

I am developing a driver for a device that exposed a class compliant midi interface, as well as vendor specific interfaces.

Sometimes after attaching / detaching the device multiple times the device will still show as attached in my test app. If I do !wdfldr I can see

    !wdfdevice 0x0000247ef9f98378 ff (FDO)
        Pnp/Power State: WdfDevStatePnpFailedWaitForRemove, WdfDevStatePowerStopped, WdfDevStatePwrPolRemoved
        context:  dt 0xffffdb8106067f70 USBDriver!DeviceContext (size is 0x60 bytes)
        EvtCleanupCallback fffff8016c289f00 USBDriver!USBDriverDeviceDestroy
        !wdfdevicequeues 0x0000247ef9f98378

The device I am working with can also be changed so that it only exposes class compliant audio and midi, and If I attach and detach the device in the mode, it will, after some time, show as selected in windows audio settings, even though the device is detached.

Everytime this happens I can see that there is a thread waiting in usbaudio!USBHwStopMIDIPipe, and it seems that it is waiting for some notification event.

0: kd> !thread ffffdb8103fe1040
THREAD ffffdb8103fe1040  Cid 0004.01f0  Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
    ffffdb8107048730  NotificationEvent
IRP List:
    ffffdb8100f5e010: (0006,04c0) Flags: 00000000  Mdl: 00000000
Not impersonating
DeviceMap                 ffff860037e478a0
Owning Process            ffffdb8ffde97200       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      602728         Ticks: 103162 (0:00:26:51.906)
Context Switch Count      241775         IdealProcessor: 1             
UserTime                  00:00:00.000
KernelTime                00:00:04.984
Win32 Start Address nt!ExpWorkerThread (0xfffff8015d48e4c0)
Stack Init ffff838fb59e7c90 Current ffff838fb59e70c0
Base ffff838fb59e8000 Limit ffff838fb59e2000 Call 0000000000000000
Priority 12 BasePriority 12 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr               : Args to Child                                                           : Call Site
ffff838f`b59e7100 fffff801`5d41c9c0     : ffffa301`bcaa0180 ffffdb81`ffffffff ffffdb81`089c5638 00000000`00000000 : nt!KiSwapContext+0x76
ffff838f`b59e7240 fffff801`5d41beef     : ffffdb81`00000001 00000000`00000000 ffff838f`b59e7400 ffffdb81`00000000 : nt!KiSwapThread+0x500
ffff838f`b59e72f0 fffff801`5d41b793     : ffffdb81`00000000 fffff801`00000000 00000000`00000000 ffffdb81`03fe1180 : nt!KiCommitThreadWait+0x14f
ffff838f`b59e7390 fffff801`6c2a3d22     : ffffdb81`07048730 00000000`00000000 ffffdb81`09cb6b00 00000000`00000000 : nt!KeWaitForSingleObject+0x233
ffff838f`b59e7480 fffff801`6c2ce3a6     : ffffdb81`0b0c3d10 ffffdb81`09cb6bf8 00000000`00000001 ffffdb81`06923240 : usbaudio!USBHwStopMIDIPipe+0x1a6
ffff838f`b59e74c0 fffff801`6c2cc5ea     : 00000000`00000000 ffffdb81`06923180 ffffdb81`09cb6bf8 ffffdb81`00000001 : usbaudio!USBHwStopAllMIDIPipes+0x42
ffff838f`b59e74f0 fffff801`6c2cc699     : 00000000`00000000 00000000`00000000 ffffdb81`060bb050 ffffdb81`09cb6b00 : usbaudio!USBDeviceStop+0x3a
ffff838f`b59e7530 fffff801`6c2c336c     : 00000000`00000000 ffffdb81`00f5e440 ffffdb81`060bb050 ffffdb81`09cb6bf8 : usbaudio!USBDeviceSurpriseRemove+0x9
ffff838f`b59e7560 fffff801`6f5a6ba3     : ffffdb81`00f5e010 ffffdb81`00f5e440 ffffdb81`0909da60 ffffdb81`0909da60 : usbaudio!DeviceSurpriseRemoval+0x20c
ffff838f`b59e75b0 ffffdb81`00f5e010     : ffffdb81`00f5e440 ffffdb81`0909da60 ffffdb81`0909da60 00000000`00000000 : 0xfffff801`6f5a6ba3
ffff838f`b59e75b8 ffffdb81`00f5e440     : ffffdb81`0909da60 ffffdb81`0909da60 00000000`00000000 00000000`00000002 : 0xffffdb81`00f5e010
ffff838f`b59e75c0 ffffdb81`0909da60     : ffffdb81`0909da60 00000000`00000000 00000000`00000002 00000000`00000800 : 0xffffdb81`00f5e440
ffff838f`b59e75c8 ffffdb81`0909da60     : 00000000`00000000 00000000`00000002 00000000`00000800 ffff6110`f7fadd09 : 0xffffdb81`0909da60
ffff838f`b59e75d0 00000000`00000000     : 00000000`00000002 00000000`00000800 ffff6110`f7fadd09 00000000`00000000 : 0xffffdb81`0909da60

0: kd> !irp ffffdb8100f5e010
Irp is active with 14 stacks 13 is current (= 0xffffdb8100f5e440)
 No Mdl: No System Buffer: Thread ffffdb8103fe1040:  Irp stack trace.  
     cmd  flg cl Device   File     Completion-Context
 [N/A(0), N/A(0)]
            0  0 00000000 00000000 00000000-00000000    

			Args: 00000000 00000000 00000000 00000000
 [N/A(0), N/A(0)]
            0  0 00000000 00000000 00000000-00000000    

			Args: 00000000 00000000 00000000 00000000
 [N/A(0), N/A(0)]
            0  0 00000000 00000000 00000000-00000000    

			Args: 00000000 00000000 00000000 00000000
 [N/A(0), N/A(0)]
            0  0 00000000 00000000 00000000-00000000    

			Args: 00000000 00000000 00000000 00000000
 [N/A(0), N/A(0)]
            0  0 00000000 00000000 00000000-00000000    

			Args: 00000000 00000000 00000000 00000000
 [N/A(0), N/A(0)]
            0  0 00000000 00000000 00000000-00000000    

			Args: 00000000 00000000 00000000 00000000
 [N/A(0), N/A(0)]
            0  0 00000000 00000000 00000000-00000000    

			Args: 00000000 00000000 00000000 00000000
 [N/A(0), N/A(0)]
            0  0 00000000 00000000 00000000-00000000    

			Args: 00000000 00000000 00000000 00000000
 [N/A(0), N/A(0)]
            0  0 00000000 00000000 00000000-00000000    

			Args: 00000000 00000000 00000000 00000000
 [N/A(0), N/A(0)]
            0  0 00000000 00000000 00000000-00000000    

			Args: 00000000 00000000 00000000 00000000
 [N/A(0), N/A(0)]
            0  0 00000000 00000000 00000000-00000000    

			Args: 00000000 00000000 00000000 00000000
 [N/A(0), N/A(0)]
            0  0 00000000 00000000 00000000-00000000    

			Args: 00000000 00000000 00000000 00000000
>[IRP_MJ_PNP(1b), IRP_MN_SURPRISE_REMOVAL(17)]
            0  0 ffffdb810909da60 00000000 00000000-00000000    
	       \Driver\usbaudio
			Args: 00000000 00000000 00000000 00000000
 [IRP_MJ_PNP(1b), IRP_MN_SURPRISE_REMOVAL(17)]
            0  0 ffffdb81072c9a70 00000000 00000000-00000000    
	       \Driver\ksthunk
			Args: 00000000 00000000 00000000 00000000

There is also another thread in usbaudio:

0: kd> !thread ffffdb8106bb8040
THREAD ffffdb8106bb8040  Cid 0004.0978  Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
    ffffdb8108cd62a0  Semaphore Limit 0x7fffffff
Not impersonating
DeviceMap                 ffff860037e478a0
Owning Process            ffffdb8ffde97200       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      602413         Ticks: 103477 (0:00:26:56.828)
Context Switch Count      1              IdealProcessor: 3             
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address usbaudio!ProcessIrpsThread (0xfffff8016c2cbb20)
Stack Init ffff838fb8a5ac90 Current ffff838fb8a5a800
Base ffff838fb8a5b000 Limit ffff838fb8a55000 Call 0000000000000000
Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr               : Args to Child                                                           : Call Site
ffff838f`b8a5a840 fffff801`5d41c9c0     : fffff801`597d8180 ffffb42f`ffffffff 00000000`00000001 00000000`00000000 : nt!KiSwapContext+0x76
ffff838f`b8a5a980 fffff801`5d41beef     : 00000001`00000000 00000000`00000003 ffff838f`b8a5ab40 ffff838f`00000000 : nt!KiSwapThread+0x500
ffff838f`b8a5aa30 fffff801`5d41b793     : 00000000`00000000 000000fa`00000000 00000000`00000000 ffffdb81`06bb8180 : nt!KiCommitThreadWait+0x14f
ffff838f`b8a5aad0 fffff801`6c2cbb63     : ffffdb81`08cd62a0 00000000`00000000 00000048`06101000 00000000`00000000 : nt!KeWaitForSingleObject+0x233
ffff838f`b8a5abc0 fffff801`5d5265f5     : ffffdb81`06bb8040 00000000`00000080 fffff801`6c2cbb20 fffff801`5d423100 : usbaudio!ProcessIrpsThread+0x43
ffff838f`b8a5ac10 fffff801`5d6048d8     : fffff801`597d8180 ffffdb81`06bb8040 fffff801`5d5265a0 ffff838f`b8a5ac80 : nt!PspSystemThreadStartup+0x55
ffff838f`b8a5ac60 00000000`00000000     : ffff838f`b8a5b000 ffff838f`b8a55000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x28

I would be very grateful if someone can point me in the right direction when it comes to debugging this. Thanks!

Output of !pnptriage mentions usbaudio!USBHwStopMIDIPipe:

https://pastebin.com/3PzGKss4

I found this post on sysnative.com forums, and the situation looks similar there. But in that case the author says that a driver update solved the issue, which does not help me so much.

0: kd> !devstack ffffdb810909da60
  !DevObj           !DrvObj            !DevExt           ObjectName
  ffffdb81072c9a70  \Driver\ksthunk    ffffdb81072c9bc0  00002454
> ffffdb810909da60  \Driver\usbaudio   ffffdb810909dbd0  
  ffffdb81089c54e0  \Driver\usbccgp    ffffdb81089c5630  00002453
!DevNode ffffdb8107590a20 :
  DeviceInst is "USB\VID_1935&PID_0B48&MI_04\9&29653da0&0&0004"
  ServiceName is "usbaudio"
0: kd> lmvm usbaudio
Browse full module list
start             end                 module name
fffff801`6c2a0000 fffff801`6c2da000   usbaudio   (pdb symbols)          C:\ProgramData\dbg\sym\USBAudio.pdb\29B99A3EFFEC8128C7AD80812C13D6AE1\USBAudio.pdb
    Loaded symbol image file: usbaudio.sys
    Image path: usbaudio.sys
    Image name: usbaudio.sys
    Browse all global symbols  functions  data
    Image was built with /Brepro flag.
    Timestamp:        F41DACD9 (This is a reproducible build file hash, not a timestamp)
    CheckSum:         0003B67F
    ImageSize:        0003A000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4
    Information from resource tables: