Device reset is happening after doing a USB selective suspend

NTDEV
1

I am doing the usb selective suspend below is the code.

     WDF_DEVICE_POWER_POLICY_IDLE_SETTINGS_INIT(&IdleSettings, IdleUsbSelectiveSuspend);
    IdleSettings.IdleTimeout = 10000;
    NtStatus = WdfDeviceAssignS0IdleSettings(Device, &IdleSettings);

    //
    // Init wait-wake policy structure.
    //
    WDF_DEVICE_POWER_POLICY_WAKE_SETTINGS_INIT(&WakeSettings);

    NtStatus = WdfDeviceAssignSxWakeSettings(Device, &WakeSettings);

I have added the above code in prepare hardware.

For the first time device goes into low power state(D2 state) and then I did some TX and device was back in D0 state.

Then I did a unplug and replug of the device and after 10 secs of being idle the device again went to low power state which is expected.
But then after some time I am getting a surprise removal event and driver gets unloaded and gets loaded again and this cycle continuous.

This is happening only if I unplug and replug the device.

Please let me what could be the cause for surprise removal and how I can fix it.