Unload issue for USBSTOR.sys

I have file system filter driver. Whenever I use self encrypting USB drive e.g. IronKey device which is composite USB device, my file system filter driver gets loaded for both the interfaces of IronKey device i.e. for CDFS as well as for Mass Storage interface.

I am facing an issue while unplugging IronKey device. Even though device is removed, sometimes it leaves Mass storage instance and later if I plug-in IronKey or any other USB device, I see yellow exclamation mark in device manager for USB Mass Storage saying “Previous instance of this device is still in memory” and that blocks USB Mass storage device enumeration. Once USBSTOR.SYS driver stops working, issue persists unless and until I reboot the machine. This is intermittent issue and not always reproducible.

My filter driver does cleanup of the instances which are created for IronKey but sometimes it fails to do so. Any suggestion to handle this situation?