Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Home NTDEV

More Info on Driver Writing and Debugging


The free OSR Learning Library has more than 50 articles on a wide variety of topics about writing and debugging device drivers and Minifilters. From introductory level to advanced. All the articles have been recently reviewed and updated, and are written using the clear and definitive style you've come to expect from OSR over the years.


Check out The OSR Learning Library at: https://www.osr.com/osr-learning-library/


Before Posting...

Please check out the Community Guidelines in the Announcements and Administration Category.

DPC CPU spikes with Isoc USB audio streaming on Windows 7 / Vista

Franz_DetroFranz_Detro Member Posts: 8
Hi,

I'm investigating a DPC CPU spike issue with lots of drivers / devices for high speed USB audio streaming.

When the devices are streaming from / to a Windows 7 or Windows Vista PC, one sees periodic DPC CPU spikes in the Windows Task Manager. Depending on the device, the spikes can use up to 100% CPU of one core for seconds. The spikes show up on both 32 and 64 bit Windows, but on the 64 bit versions they are typically spreaded amongst all cores. On Windows XP this problem does not exist, no spikes at all.

The driver that I am maintaining works with a continous Isoc reader that schedules 4 - 8 URBs of 8 packets (one per micro frame) with 512 byte capacity.

I've tried to remove all productive code which may delay the URB completition, but the problem persists even if only the input data is read from the device (no processing at all).

Is this a known issue with Windows 7 / Windows Vista, or are there some special tricks to get Isoc high speed streaming working reliable there?

Regards,
Franz

Comments

  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    Have you tried disabling other hardware? Especially network adapter
    driver turn out to be source for DPC holidays. Other than that there
    are a some measurement tools indicating DPC caused latency issues you
    can use to find the assaulting driver.

    Best,
    Hagen.

    On Mon, Jun 7, 2010 at 4:10 PM, Franz Detro
    <[email protected]> wrote:
    > Hi,
    >
    > I'm investigating a DPC CPU spike issue with lots of drivers / devices for high speed USB audio streaming.
    >
    > When the devices are streaming from / to a Windows 7 or Windows Vista PC, one sees periodic DPC CPU spikes in the Windows Task Manager. Depending on the device, the spikes can use up to 100% CPU of one core for seconds. The spikes show up on both 32 and 64 bit Windows, but on the 64 bit versions they are typically spreaded amongst all cores. On Windows XP this problem does not exist, no spikes at all.
    >
    > The driver that I am maintaining works with a continous Isoc reader that schedules 4 - 8 URBs of 8 packets (one per micro frame) with 512 byte capacity.
    >
    > I've tried to remove all productive code which may delay the URB completition, but the problem persists even if only the input data is read from the device (no processing at all).
    >
    > Is this a known issue with Windows 7 / Windows Vista, or are there some special tricks to get Isoc high speed streaming working reliable there?
    >
    > Regards,
    > Franz
    >
    >
    > ---
    > NTDEV is sponsored by OSR
    >
    > For our schedule of WDF, WDM, debugging and other seminars visit:
    > http://www.osr.com/seminars
    >
    > To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
    >
  • Franz_DetroFranz_Detro Member Posts: 8
    Hi,

    > Have you tried disabling other hardware? Especially network adapter
    > driver turn out to be source for DPC holidays.

    yes. I've tried to disable network, inboard sound cards, even unused internal USB hubs, no difference.

    > Other than that there
    > are a some measurement tools indicating DPC caused latency issues you
    > can use to find the assaulting driver.

    I've used the Thesycon DPC latency checker, which does not show any spikes.

    I did traces with the Windows 7 Performance Analyzer, which points me to USBPORT.SYS::USBPORT_Xdpc_Worker.

    80 % of the calls are spending max. 0.066 ms there (20 % only 0.007 ms), but the remaining 20 % do spend up to 0.253 ms there. One remarkable thing is that there is a 'gap' in the samples between 0.066 ms and 0.105 ms. In other words: it looks like there is some special condition which causes the DPC to use 2 - 5 times more CPU than the 'maximal standard case'.

    All my analysis points to a problem with the Windows 7 / Vista implementation of USBPORT.SYS.

    >
    > Best,
    > Hagen.
    >

    Best,
    Franz
  • Philip_RiesPhilip_Ries Member Posts: 64
    There is a related known issue in Win7, with a hotfix available:

    http://support.microsoft.com/kb/981214

    Franz Detro wrote:
    > Hi,
    >
    >> Have you tried disabling other hardware? Especially network adapter
    >> driver turn out to be source for DPC holidays.
    >
    > yes. I've tried to disable network, inboard sound cards, even unused internal USB hubs, no difference.
    >
    >> Other than that there
    >> are a some measurement tools indicating DPC caused latency issues you
    >> can use to find the assaulting driver.
    >
    > I've used the Thesycon DPC latency checker, which does not show any spikes.
    >
    > I did traces with the Windows 7 Performance Analyzer, which points me to USBPORT.SYS::USBPORT_Xdpc_Worker.
    >
    > 80 % of the calls are spending max. 0.066 ms there (20 % only 0.007 ms), but the remaining 20 % do spend up to 0.253 ms there. One remarkable thing is that there is a 'gap' in the samples between 0.066 ms and 0.105 ms. In other words: it looks like there is some special condition which causes the DPC to use 2 - 5 times more CPU than the 'maximal standard case'.
    >
    > All my analysis points to a problem with the Windows 7 / Vista implementation of USBPORT.SYS.
    >
    >> Best,
    >> Hagen.
    >>
    >
    > Best,
    > Franz
    >
    >
  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 8,483
    You need to do some analysis to find out which driver is causing the long DPCs.

    Try using XPerf:

    http://www.osronline.com/article.cfm?article=554

    Peter
    OSR

    Peter Viscarola
    OSR
    @OSRDrivers

  • Franz_DetroFranz_Detro Member Posts: 8
    > There is a related known issue in Win7, with a hotfix available:
    >
    > http://support.microsoft.com/kb/981214

    Thanks, the problem description is pretty close to my issue. I've installed this fix, but now it's even worse: CPU spikes on *all* cores, more audio dropouts than before.

    Nevertheless this is a hint pointing straight to a USBPORT.SYS problem, I will try to contact MS support.

    Thanks,
    Franz

    >
    > Franz Detro wrote:
    >> Hi,
    >>> Have you tried disabling other hardware? Especially network adapter
    >>> driver turn out to be source for DPC holidays.
    >> yes. I've tried to disable network, inboard sound cards, even unused internal USB hubs, no difference.
    >>> Other than that there
    >>> are a some measurement tools indicating DPC caused latency issues you
    >>> can use to find the assaulting driver.
    >> I've used the Thesycon DPC latency checker, which does not show any spikes.
    >> I did traces with the Windows 7 Performance Analyzer, which points me to USBPORT.SYS::USBPORT_Xdpc_Worker.
    >> 80 % of the calls are spending max. 0.066 ms there (20 % only 0.007 ms), but the remaining 20 % do spend up to 0.253 ms there. One remarkable thing is that there is a 'gap' in the samples between 0.066 ms and 0.105 ms. In other words: it looks like there is some special condition which causes the DPC to use 2 - 5 times more CPU than the 'maximal standard case'.
    >> All my analysis points to a problem with the Windows 7 / Vista implementation of USBPORT.SYS.
    >>> Best,
    >>> Hagen.
    >>>
    >> Best,
    >> Franz
    >
    > ---
    > NTDEV is sponsored by OSR
    >
    > For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars
    >
    > To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
  • Franz_DetroFranz_Detro Member Posts: 8
    Hi Peter,

    > You need to do some analysis to find out which driver is causing the long DPCs.
    >
    > Try using XPerf:
    >
    > http://www.osronline.com/article.cfm?article=554

    I did this, and it points me to USBPORT.SYS::USBPORT.SYS::USBPORT_Xdpc_Worker. I've posted some results in another mail of this thread.

    >
    > Peter
    > OSR

    Thanks,
    Franz

    >
    >
    > ---
    > NTDEV is sponsored by OSR
    >
    > For our schedule of WDF, WDM, debugging and other seminars visit:
    > http://www.osr.com/seminars
    >
    > To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 8,483
    But your previous post said you were seeing MAXIMUM 250us spikes.

    That's not tiny, but it really doesn't seem like an amount of time that would cause problems either. Are you giving us a complete picture of your issue?

    Or, perhaps, you really did just want to know if this was a known problem, and the answers we've given you so far are sufficient?

    Peter
    OSR

    Peter Viscarola
    OSR
    @OSRDrivers

  • Franz_DetroFranz_Detro Member Posts: 8
    Hi Peter,

    >
    > But your previous post said you were seeing MAXIMUM 250us spikes.

    yes, exactely. And this is confusing me:

    - the Windows Task manager shows periodic DPC / Kernel CPU usage spikes of up to 100 % every 10 - 15 seconds while streaming audio.

    - the xperf log shows DPC times between nearly 0 and 0,25 ms

    I know that these pictures do not fit together, but that's all I have.

    >
    > That's not tiny, but it really doesn't seem like an amount of time that would cause problems either.

    I'm not so sure about this. 0.25 ms URB DPC time for a URB once a ms? This means that the system spends 25% (!!) of one core only in this DPC?

    On Windows XP, the overall DPC / kernel CPU usage is much smaller (2 - 3%) than on Vista and Win 7, and no spikes.

    Unfortunaly, xperf does not work on Windows XP, as I know, so I'm not able to get the corresponding XP numbers.

    > Are you giving us a complete picture of your issue?

    I'm trying. I can send you some screenshots and the xperf log files, if you want.

    >
    > Or, perhaps, you really did just want to know if this was a known problem,

    Partially, yes. Or if not, what one can do to avoid these spikes / what our drivers are doing wrong.

    > and the answers we've given you so far are sufficient?

    Unfortunaly not. The Hotfix issue http://support.microsoft.com/kb/981214 looks pretty much like our problem, but applying the hotfix to my system made things even worse.

    Sorry for not giving you more precise information, I'm still trying to get some light into this problem.

    >
    > Peter
    > OSR

    Best,
    Franz

    >
    >
    > ---
    > NTDEV is sponsored by OSR
    >
    > For our schedule of WDF, WDM, debugging and other seminars visit:
    > http://www.osr.com/seminars
    >
    > To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
  • Udo_EberhardtUdo_Eberhardt Member Posts: 50
    Hi Franz,

    I have the same problem with my driver. My driver gets a URB completion (DPC) every 4 millisecs. When isoch streaming is running I'm seeing the same problem: periodically one core reaches 100% load for ~2 seconds. The period is approximately 20 seconds.

    I did also try the hot fix posted here. There was no significant change. The problem still persists.

    I analyzed the problem using Intel VTune. VTune shows that when the core reaches 100% then 90% of the total amount of CPU cycles are consumed by the Microsoft usbport/usbehci driver. Only 10% are consumed by my driver. Further, VTune indicates that the most amount of time is spent in a routine named EHCI_InternalPollHsIsoEndpoint which is in usbehci.sys. This routine seems to spin in a loop (as its name implies) and repeatedly call READ_REGISTER_ULONG and InterlockedDecrement. So the issue seems to be caused by the the EHCI driver implementation. As this problem does not exist on XP (while isoch streaming is working equally well) it should be possible to fix it.
  • Philip_RiesPhilip_Ries Member Posts: 64
    It sounds like you guys are hitting issues that need to be investigated
    more deeply. For information on how to open a Windows Driver Kit (WDK)
    service request in order to receive USB driver development support,
    please click on the below link:

    http://support.microsoft.com/select/?LN=en-us&target=assistance&x=13&y=9

    In the “Quick product finder” field, type “Windows Driver Kit” as the
    product and select that product.
    Thanks,
    Philip

    [email protected] wrote:
    > Hi Franz,
    >
    > I have the same problem with my driver. My driver gets a URB completion (DPC) every 4 millisecs. When isoch streaming is running I'm seeing the same problem: periodically one core reaches 100% load for ~2 seconds. The period is approximately 20 seconds.
    >
    > I did also try the hot fix posted here. There was no significant change. The problem still persists.
    >
    > I analyzed the problem using Intel VTune. VTune shows that when the core reaches 100% then 90% of the total amount of CPU cycles are consumed by the Microsoft usbport/usbehci driver. Only 10% are consumed by my driver. Further, VTune indicates that the most amount of time is spent in a routine named EHCI_InternalPollHsIsoEndpoint which is in usbehci.sys. This routine seems to spin in a loop (as its name implies) and repeatedly call READ_REGISTER_ULONG and InterlockedDecrement. So the issue seems to be caused by the the EHCI driver implementation. As this problem does not exist on XP (while isoch streaming is working equally well) it should be possible to fix it.
    >
    >
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    Hi Franz, I'm one of the developers in Windows who worked on the KB mentioned in this article. Can you answer a few questions for me?

    * Is the audio device you're using connected directly to the PC, or connected via a hub? If it's the latter, do you still see problems when directly connected?

    * Is there any shipping NI hardware that reproduces this problem?

    * Can you send us the XPerf trace that you took?
  • Franz_DetroFranz_Detro Member Posts: 8
    Hi,

    > Hi Franz, I'm one of the developers in Windows who worked on the KB mentioned in this article. Can you answer a few questions for me?

    sure.

    >
    > * Is the audio device you're using connected directly to the PC, or connected via a hub? If it's the latter, do you still see problems when directly connected?

    The devices are connected directly to the PC.

    >
    > * Is there any shipping NI hardware that reproduces this problem?

    Yes. You can reproduce this with every NI interface out there.

    >
    > * Can you send us the XPerf trace that you took?

    Yes. I will send you a private mail.

    Best,
    Franz
Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Upcoming OSR Seminars
OSR has suspended in-person seminars due to the Covid-19 outbreak. But, don't miss your training! Attend via the internet instead!
Developing Minifilters 24 May 2021 Live, Online
Writing WDF Drivers 14 June 2021 Live, Online
Internals & Software Drivers 27 September 2021 Live, Online
Kernel Debugging 15 November 2021 Live, Online