DPC CPU spikes with Isoc USB audio streaming on Windows 7 / Vista

Hi,

I’m investigating a DPC CPU spike issue with lots of drivers / devices for high speed USB audio streaming.

When the devices are streaming from / to a Windows 7 or Windows Vista PC, one sees periodic DPC CPU spikes in the Windows Task Manager. Depending on the device, the spikes can use up to 100% CPU of one core for seconds. The spikes show up on both 32 and 64 bit Windows, but on the 64 bit versions they are typically spreaded amongst all cores. On Windows XP this problem does not exist, no spikes at all.

The driver that I am maintaining works with a continous Isoc reader that schedules 4 - 8 URBs of 8 packets (one per micro frame) with 512 byte capacity.

I’ve tried to remove all productive code which may delay the URB completition, but the problem persists even if only the input data is read from the device (no processing at all).

Is this a known issue with Windows 7 / Windows Vista, or are there some special tricks to get Isoc high speed streaming working reliable there?

Regards,
Franz

Have you tried disabling other hardware? Especially network adapter
driver turn out to be source for DPC holidays. Other than that there
are a some measurement tools indicating DPC caused latency issues you
can use to find the assaulting driver.

Best,
Hagen.

On Mon, Jun 7, 2010 at 4:10 PM, Franz Detro
wrote:
> Hi,
>
> I’m investigating a DPC CPU spike issue with lots of drivers / devices for high speed USB audio streaming.
>
> When the devices are streaming from / to a Windows 7 or Windows Vista PC, one sees periodic DPC CPU spikes in the Windows Task Manager. Depending on the device, the spikes can use up to 100% CPU of one core for seconds. The spikes show up on both 32 and 64 bit Windows, but on the 64 bit versions they are typically spreaded amongst all cores. On Windows XP this problem does not exist, no spikes at all.
>
> The driver that I am maintaining works with a continous Isoc reader that schedules 4 - 8 URBs of 8 packets (one per micro frame) with 512 byte capacity.
>
> I’ve tried to remove all productive code which may delay the URB completition, but the problem persists even if only the input data is read from the device (no processing at all).
>
> Is this a known issue with Windows 7 / Windows Vista, or are there some special tricks to get Isoc high speed streaming working reliable there?
>
> Regards,
> Franz
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>

Hi,

Have you tried disabling other hardware? Especially network adapter
driver turn out to be source for DPC holidays.

yes. I’ve tried to disable network, inboard sound cards, even unused internal USB hubs, no difference.

Other than that there
are a some measurement tools indicating DPC caused latency issues you
can use to find the assaulting driver.

I’ve used the Thesycon DPC latency checker, which does not show any spikes.

I did traces with the Windows 7 Performance Analyzer, which points me to USBPORT.SYS::USBPORT_Xdpc_Worker.

80 % of the calls are spending max. 0.066 ms there (20 % only 0.007 ms), but the remaining 20 % do spend up to 0.253 ms there. One remarkable thing is that there is a ‘gap’ in the samples between 0.066 ms and 0.105 ms. In other words: it looks like there is some special condition which causes the DPC to use 2 - 5 times more CPU than the ‘maximal standard case’.

All my analysis points to a problem with the Windows 7 / Vista implementation of USBPORT.SYS.

Best,
Hagen.

Best,
Franz

There is a related known issue in Win7, with a hotfix available:

http://support.microsoft.com/kb/981214

Franz Detro wrote:

Hi,

> Have you tried disabling other hardware? Especially network adapter
> driver turn out to be source for DPC holidays.

yes. I’ve tried to disable network, inboard sound cards, even unused internal USB hubs, no difference.

> Other than that there
> are a some measurement tools indicating DPC caused latency issues you
> can use to find the assaulting driver.

I’ve used the Thesycon DPC latency checker, which does not show any spikes.

I did traces with the Windows 7 Performance Analyzer, which points me to USBPORT.SYS::USBPORT_Xdpc_Worker.

80 % of the calls are spending max. 0.066 ms there (20 % only 0.007 ms), but the remaining 20 % do spend up to 0.253 ms there. One remarkable thing is that there is a ‘gap’ in the samples between 0.066 ms and 0.105 ms. In other words: it looks like there is some special condition which causes the DPC to use 2 - 5 times more CPU than the ‘maximal standard case’.

All my analysis points to a problem with the Windows 7 / Vista implementation of USBPORT.SYS.

> Best,
> Hagen.
>

Best,
Franz

You need to do some analysis to find out which driver is causing the long DPCs.

Try using XPerf:

http://www.osronline.com/article.cfm?article=554

Peter
OSR

> There is a related known issue in Win7, with a hotfix available:

http://support.microsoft.com/kb/981214

Thanks, the problem description is pretty close to my issue. I’ve installed this fix, but now it’s even worse: CPU spikes on *all* cores, more audio dropouts than before.

Nevertheless this is a hint pointing straight to a USBPORT.SYS problem, I will try to contact MS support.

Thanks,
Franz

Franz Detro wrote:
> Hi,
>> Have you tried disabling other hardware? Especially network adapter
>> driver turn out to be source for DPC holidays.
> yes. I’ve tried to disable network, inboard sound cards, even unused internal USB hubs, no difference.
>> Other than that there
>> are a some measurement tools indicating DPC caused latency issues you
>> can use to find the assaulting driver.
> I’ve used the Thesycon DPC latency checker, which does not show any spikes.
> I did traces with the Windows 7 Performance Analyzer, which points me to USBPORT.SYS::USBPORT_Xdpc_Worker.
> 80 % of the calls are spending max. 0.066 ms there (20 % only 0.007 ms), but the remaining 20 % do spend up to 0.253 ms there. One remarkable thing is that there is a ‘gap’ in the samples between 0.066 ms and 0.105 ms. In other words: it looks like there is some special condition which causes the DPC to use 2 - 5 times more CPU than the ‘maximal standard case’.
> All my analysis points to a problem with the Windows 7 / Vista implementation of USBPORT.SYS.
>> Best,
>> Hagen.
>>
> Best,
> Franz


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Hi Peter,

You need to do some analysis to find out which driver is causing the long DPCs.

Try using XPerf:

http://www.osronline.com/article.cfm?article=554

I did this, and it points me to USBPORT.SYS::USBPORT.SYS::USBPORT_Xdpc_Worker. I’ve posted some results in another mail of this thread.

Peter
OSR

Thanks,
Franz


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

But your previous post said you were seeing MAXIMUM 250us spikes.

That’s not tiny, but it really doesn’t seem like an amount of time that would cause problems either. Are you giving us a complete picture of your issue?

Or, perhaps, you really did just want to know if this was a known problem, and the answers we’ve given you so far are sufficient?

Peter
OSR

Hi Peter,

But your previous post said you were seeing MAXIMUM 250us spikes.

yes, exactely. And this is confusing me:

  • the Windows Task manager shows periodic DPC / Kernel CPU usage spikes of up to 100 % every 10 - 15 seconds while streaming audio.

  • the xperf log shows DPC times between nearly 0 and 0,25 ms

I know that these pictures do not fit together, but that’s all I have.

That’s not tiny, but it really doesn’t seem like an amount of time that would cause problems either.

I’m not so sure about this. 0.25 ms URB DPC time for a URB once a ms? This means that the system spends 25% (!!) of one core only in this DPC?

On Windows XP, the overall DPC / kernel CPU usage is much smaller (2 - 3%) than on Vista and Win 7, and no spikes.

Unfortunaly, xperf does not work on Windows XP, as I know, so I’m not able to get the corresponding XP numbers.

Are you giving us a complete picture of your issue?

I’m trying. I can send you some screenshots and the xperf log files, if you want.

Or, perhaps, you really did just want to know if this was a known problem,

Partially, yes. Or if not, what one can do to avoid these spikes / what our drivers are doing wrong.

and the answers we’ve given you so far are sufficient?

Unfortunaly not. The Hotfix issue http://support.microsoft.com/kb/981214 looks pretty much like our problem, but applying the hotfix to my system made things even worse.

Sorry for not giving you more precise information, I’m still trying to get some light into this problem.

Peter
OSR

Best,
Franz


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Hi Franz,

I have the same problem with my driver. My driver gets a URB completion (DPC) every 4 millisecs. When isoch streaming is running I’m seeing the same problem: periodically one core reaches 100% load for ~2 seconds. The period is approximately 20 seconds.

I did also try the hot fix posted here. There was no significant change. The problem still persists.

I analyzed the problem using Intel VTune. VTune shows that when the core reaches 100% then 90% of the total amount of CPU cycles are consumed by the Microsoft usbport/usbehci driver. Only 10% are consumed by my driver. Further, VTune indicates that the most amount of time is spent in a routine named EHCI_InternalPollHsIsoEndpoint which is in usbehci.sys. This routine seems to spin in a loop (as its name implies) and repeatedly call READ_REGISTER_ULONG and InterlockedDecrement. So the issue seems to be caused by the the EHCI driver implementation. As this problem does not exist on XP (while isoch streaming is working equally well) it should be possible to fix it.

It sounds like you guys are hitting issues that need to be investigated
more deeply. For information on how to open a Windows Driver Kit (WDK)
service request in order to receive USB driver development support,
please click on the below link:

http://support.microsoft.com/select/?LN=en-us&target=assistance&x=13&y=9

In the “Quick product finder” field, type “Windows Driver Kit” as the
product and select that product.
Thanks,
Philip

xxxxx@thesycon.de wrote:

Hi Franz,

I have the same problem with my driver. My driver gets a URB completion (DPC) every 4 millisecs. When isoch streaming is running I’m seeing the same problem: periodically one core reaches 100% load for ~2 seconds. The period is approximately 20 seconds.

I did also try the hot fix posted here. There was no significant change. The problem still persists.

I analyzed the problem using Intel VTune. VTune shows that when the core reaches 100% then 90% of the total amount of CPU cycles are consumed by the Microsoft usbport/usbehci driver. Only 10% are consumed by my driver. Further, VTune indicates that the most amount of time is spent in a routine named EHCI_InternalPollHsIsoEndpoint which is in usbehci.sys. This routine seems to spin in a loop (as its name implies) and repeatedly call READ_REGISTER_ULONG and InterlockedDecrement. So the issue seems to be caused by the the EHCI driver implementation. As this problem does not exist on XP (while isoch streaming is working equally well) it should be possible to fix it.

Hi Franz, I’m one of the developers in Windows who worked on the KB mentioned in this article. Can you answer a few questions for me?

* Is the audio device you’re using connected directly to the PC, or connected via a hub? If it’s the latter, do you still see problems when directly connected?

* Is there any shipping NI hardware that reproduces this problem?

* Can you send us the XPerf trace that you took?

Hi,

Hi Franz, I’m one of the developers in Windows who worked on the KB mentioned in this article. Can you answer a few questions for me?

sure.

* Is the audio device you’re using connected directly to the PC, or connected via a hub? If it’s the latter, do you still see problems when directly connected?

The devices are connected directly to the PC.

* Is there any shipping NI hardware that reproduces this problem?

Yes. You can reproduce this with every NI interface out there.

* Can you send us the XPerf trace that you took?

Yes. I will send you a private mail.

Best,
Franz