How does MSFT debug mysterious code in Windows?

Hi,

During Windows (NT 6 and above) boots,there are certain time windows that KD
is shutdown. For the sake of discussion, let

t0 = the time BIOS or EFI hands over control to bootmgr
t1=bootmgr switches CPU to protected mode with paging off.
t2=bootmgr turns on boot KD.
t3=bootmgr turned off boot KD and hands over to winload/winresume.
t4=winload/winresume turns on boot KD
t5=winload/winresume turned off boot KD and hands over to ntoskrnl/hal
t6=ntos turns on normal KD.

Most of debugging activities are carried over during the time of t>t6 so
life is good. I was unfortunate enough having to debug during open intervals
of (t0, t2), (t3, t4) and (t5, t6) because Windows hangs during one of those
intervals while booting from my SAN/Converged NICs. KD is off so I resorted
to using an ICE. ICE is a great (and expensive) tool but it has
shortcomings, i.e. not many system has JTAG/XDP ports. Have to get help from
OEM to populate the port on MB if possible at all; ICE doesn’t have Windows
NT specific knowledges, a simple task in kd can be a big monster to attack
in ICE environment.

There are a lot of code being executed in those “dark” areas. How do MSFT
engineers or other folks debug that piece of code in a more Windows friendly
way?

Best regards,
Calvin

An ICE is a PITA, I agree. That’s what we use for the sort of stuff you describe, mostly, though. We’re lucky enough that – so far at least – the motherboard on which we’re running doesn’t matter to the project. So for us we can setup the ICE in ONE configuration with an appropriate interposer, or by grabbing any old crufty last-year’s CRB (which always seem to have XDP ports) and using that.

If you’re sufficiently patient, you can boot under Bochs (http://bochs.sourceforge.net/) and use their debugger. But you DO have to be patient. We recently did use Bochs for a project, as opposed to our Arium, and it wasn’t nearly as slow as we thought it would be.

I don’t know if that’s any help but I hope so,

Peter
OSR

Hi Peter,

Thank you for sharing your debugging experiences. It’s good to know that I’m
not the one that have to gone through this…

bochs is a great thing. I used it to developed my boot code that doesn’t
depend on specific hardware. In the Windows boot scenario I described, my
ROM code on my SAN/Converged NIC adapter needs to fetch boot files from SAN
fabric on behalf of bootmgr/winload. I’m not sure if bochs supports such
environment.

Thanks,
Calvin
On Fri, Oct 14, 2011 at 9:45 AM, wrote:

> An ICE is a PITA, I agree. That’s what we use for the sort of stuff you
> describe, mostly, though. We’re lucky enough that – so far at least – the
> motherboard on which we’re running doesn’t matter to the project. So for us
> we can setup the ICE in ONE configuration with an appropriate interposer, or
> by grabbing any old crufty last-year’s CRB (which always seem to have XDP
> ports) and using that.
>
> If you’re sufficiently patient, you can boot under Bochs (
> http://bochs.sourceforge.net/) and use their debugger. But you DO have to
> be patient. We recently did use Bochs for a project, as opposed to our
> Arium, and it wasn’t nearly as slow as we thought it would be.
>
> I don’t know if that’s any help but I hope so,
>
> Peter
> OSR
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

Unfortunately, I don’t have much to offer here that Peter hasn’t already
covered. While the Arium can be occasionally be hugely helpful, in general,
even on the best of days, on systems to which you actually can attach,
they’re wonky.

mm

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Calvin Guan (news)
Sent: Friday, October 14, 2011 10:06 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] How does MSFT debug mysterious code in Windows?

Hi Peter,

Thank you for sharing your debugging experiences. It’s good to know that I’m
not the one that have to gone through this…

bochs is a great thing. I used it to developed my boot code that doesn’t
depend on specific hardware. In the Windows boot scenario I described, my
ROM code on my SAN/Converged NIC adapter needs to fetch boot files from SAN
fabric on behalf of bootmgr/winload. I’m not sure if bochs supports such
environment.

Thanks,

Calvin

On Fri, Oct 14, 2011 at 9:45 AM, wrote:

An ICE is a PITA, I agree. That’s what we use for the sort of stuff you
describe, mostly, though. We’re lucky enough that – so far at least – the
motherboard on which we’re running doesn’t matter to the project. So for us
we can setup the ICE in ONE configuration with an appropriate interposer, or
by grabbing any old crufty last-year’s CRB (which always seem to have XDP
ports) and using that.

If you’re sufficiently patient, you can boot under Bochs
(http://bochs.sourceforge.net/) and use their debugger. But you DO have to
be patient. We recently did use Bochs for a project, as opposed to our
Arium, and it wasn’t nearly as slow as we thought it would be.

I don’t know if that’s any help but I hope so,

Peter
OSR

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and
other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the
List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

I agreed with you guys with regards to the effectiveness and pain of using
an ICE debugging Windows. It’s kind of like solving 100 And yes, I
keep various models of Arium for different generations of CPU/chipset.
They are the only ones to save my day once in a while.

Thanks,
Calvin

On Fri, Oct 14, 2011 at 1:02 PM, Martin O’Brien <
xxxxx@gmail.com> wrote:

Unfortunately, I don?t have much to offer here that Peter hasn?t already
covered. While the Arium can be occasionally be hugely helpful, in general,
even on the best of days, on systems to which you actually can attach,
they?re wonky.****

** **

** **

mm****

** **

*From:* xxxxx@lists.osr.com [mailto:
xxxxx@lists.osr.com] *On Behalf Of *Calvin Guan (news)
*Sent:* Friday, October 14, 2011 10:06 AM
*To:* Windows System Software Devs Interest List
*Subject:* Re: [ntdev] How does MSFT debug mysterious code in Windows?****

** **

Hi Peter,****

****

Thank you for sharing your debugging experiences. It’s good to know that
I’m not the one that have to gone through this…****

****

bochs is a great thing. I used it to developed my boot code that doesn’t
depend on specific hardware. In the Windows boot scenario I described, my
ROM code on my SAN/Converged NIC adapter needs to fetch boot files from SAN
fabric on behalf of bootmgr/winload. I’m not sure if bochs supports such
environment.****

****

Thanks,****

Calvin****

On Fri, Oct 14, 2011 at 9:45 AM, wrote:
>
> An ICE is a PITA, I agree. That’s what we use for the sort of stuff you
> describe, mostly, though. We’re lucky enough that – so far at least – the
> motherboard on which we’re running doesn’t matter to the project. So for us
> we can setup the ICE in ONE configuration with an appropriate interposer, or
> by grabbing any old crufty last-year’s CRB (which always seem to have XDP
> ports) and using that.
>
> If you’re sufficiently patient, you can boot under Bochs (
> http://bochs.sourceforge.net/) and use their debugger. But you DO have to
> be patient. We recently did use Bochs for a project, as opposed to our
> Arium, and it wasn’t nearly as slow as we thought it would be.
>
> I don’t know if that’s any help but I hope so,
>
> Peter
> OSR
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and
> other seminars visit: http://www.osr.com/seminars To unsubscribe, visit
> the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer ****
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

That’s the thing - every once in a blue moon, they save me a huge amount of
time. Otherwise, they are one of the biggest PITA’s going, I think, though
I’d say that’s really more due to Intel than Arium.

mm

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Calvin Guan (news)
Sent: Friday, October 14, 2011 7:02 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] How does MSFT debug mysterious code in Windows?

I agreed with you guys with regards to the effectiveness and pain of using
an ICE debugging Windows. It’s kind of like solving 100 And yes, I keep
various models of Arium for different generations of CPU/chipset. They are
the only ones to save my day once in a while.

Thanks,

Calvin

On Fri, Oct 14, 2011 at 1:02 PM, Martin O’Brien
wrote:

Unfortunately, I don’t have much to offer here that Peter hasn’t already
covered. While the Arium can be occasionally be hugely helpful, in general,
even on the best of days, on systems to which you actually can attach,
they’re wonky.

mm

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Calvin Guan (news)
Sent: Friday, October 14, 2011 10:06 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] How does MSFT debug mysterious code in Windows?

Hi Peter,

Thank you for sharing your debugging experiences. It’s good to know that I’m
not the one that have to gone through this…

bochs is a great thing. I used it to developed my boot code that doesn’t
depend on specific hardware. In the Windows boot scenario I described, my
ROM code on my SAN/Converged NIC adapter needs to fetch boot files from SAN
fabric on behalf of bootmgr/winload. I’m not sure if bochs supports such
environment.

Thanks,

Calvin

On Fri, Oct 14, 2011 at 9:45 AM, wrote:

An ICE is a PITA, I agree. That’s what we use for the sort of stuff you
describe, mostly, though. We’re lucky enough that – so far at least – the
motherboard on which we’re running doesn’t matter to the project. So for us
we can setup the ICE in ONE configuration with an appropriate interposer, or
by grabbing any old crufty last-year’s CRB (which always seem to have XDP
ports) and using that.

If you’re sufficiently patient, you can boot under Bochs
(http://bochs.sourceforge.net/) and use their debugger. But you DO have to
be patient. We recently did use Bochs for a project, as opposed to our
Arium, and it wasn’t nearly as slow as we thought it would be.

I don’t know if that’s any help but I hope so,

Peter
OSR

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and
other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the
List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and
other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the
List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Can’t blame Arium. I am happily using it to debug my EFI drivers and pre-os
bootcode on a daily basis. If somebody somehow ports the the KD environment
to Arium, that would sell:)

Calvin
On Fri, Oct 14, 2011 at 7:28 PM, Martin O’Brien <
xxxxx@gmail.com> wrote:

That?s the thing ? every once in a blue moon, they save me a huge amount of
time. Otherwise, they are one of the biggest PITA?s going, I think, though
I?d say that?s really more due to Intel than Arium.****

** **

** **

mm****

** **

*From:* xxxxx@lists.osr.com [mailto:
xxxxx@lists.osr.com] *On Behalf Of *Calvin Guan (news)
*Sent:* Friday, October 14, 2011 7:02 PM

*To:* Windows System Software Devs Interest List
*Subject:* Re: [ntdev] How does MSFT debug mysterious code in Windows?****

** **

I agreed with you guys with regards to the effectiveness and pain of using
an ICE debugging Windows. It’s kind of like solving 100 And yes, I
keep various models of Arium for different generations of CPU/chipset.
They are the only ones to save my day once in a while.****

****

Thanks,****

Calvin****

On Fri, Oct 14, 2011 at 1:02 PM, Martin O’Brien <
xxxxx@gmail.com> wrote:****

Unfortunately, I don?t have much to offer here that Peter hasn?t already
covered. While the Arium can be occasionally be hugely helpful, in general,
even on the best of days, on systems to which you actually can attach,
they?re wonky.****

****

****

mm****

****

*From:* xxxxx@lists.osr.com [mailto:
xxxxx@lists.osr.com] *On Behalf Of *Calvin Guan (news)
*Sent:* Friday, October 14, 2011 10:06 AM
*To:* Windows System Software Devs Interest List
*Subject:* Re: [ntdev] How does MSFT debug mysterious code in Windows?****

****

Hi Peter,****

****

Thank you for sharing your debugging experiences. It’s good to know that
I’m not the one that have to gone through this…****

****

bochs is a great thing. I used it to developed my boot code that doesn’t
depend on specific hardware. In the Windows boot scenario I described, my
ROM code on my SAN/Converged NIC adapter needs to fetch boot files from SAN
fabric on behalf of bootmgr/winload. I’m not sure if bochs supports such
environment.****

****

Thanks,****

Calvin****

On Fri, Oct 14, 2011 at 9:45 AM, wrote:
>
> An ICE is a PITA, I agree. That’s what we use for the sort of stuff you
> describe, mostly, though. We’re lucky enough that – so far at least – the
> motherboard on which we’re running doesn’t matter to the project. So for us
> we can setup the ICE in ONE configuration with an appropriate interposer, or
> by grabbing any old crufty last-year’s CRB (which always seem to have XDP
> ports) and using that.
>
> If you’re sufficiently patient, you can boot under Bochs (
> http://bochs.sourceforge.net/) and use their debugger. But you DO have to
> be patient. We recently did use Bochs for a project, as opposed to our
> Arium, and it wasn’t nearly as slow as we thought it would be.
>
> I don’t know if that’s any help but I hope so,
>
> Peter
> OSR
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and
> other seminars visit: http://www.osr.com/seminars To unsubscribe, visit
> the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and
> other seminars visit: http://www.osr.com/seminars To unsubscribe, visit
> the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer ****
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

Sadly, I think it would sell about 5 copies.

There aren’t that many of us who have Ariums, I bet. Even fewer are used by “software” people to debug drivers. Which is too bad, because if more people DID use these devices there WOULD be better tools for using them the way we do.

In my experience, if a driver dev has even HEARD of an Arium, they seem to recall that they’re “very very very expensive”, when in fact that’s no longer the case… they’re merely “very expensive”

Peter
OSR

wrote in message news:xxxxx@ntdev…
…
> In my experience, if a driver dev has even HEARD of an Arium, they seem to
> recall that they’re “very very very expensive”, when in fact that’s no
> longer the case… they’re merely “very expensive”

Development since then has moved to places where it’s still 3*very, if not
4.
– pa

And a few ‘very’-s have gotten prepended to the part about getting a useful
CRB board from Intel, in my opinion.

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Pavel A.
Sent: Saturday, October 15, 2011 8:25 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] How does MSFT debug mysterious code in Windows?

wrote in message news:xxxxx@ntdev…
…
> In my experience, if a driver dev has even HEARD of an Arium, they
> seem to recall that they’re “very very very expensive”, when in fact
> that’s no longer the case… they’re merely “very expensive”

Development since then has moved to places where it’s still 3*very, if not
4.
– pa

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Comparing to a gen2 PCIe exerciser, Synopsys, Cadence, a 3-D full wave
Maxwell Solver, the server farm to crunch the numbers, or some of the funny
“toys” in the lab, etc etc… it’s bearable… Not counting the cost of taping
out a chip yet…

Making a chip and selling it at a fraction of the price of a golf club
appears to be a fool’s game.

Calvin
On Sat, Oct 15, 2011 at 8:35 PM, Martin O’Brien <
xxxxx@gmail.com> wrote:

And a few ‘very’-s have gotten prepended to the part about getting a useful
CRB board from Intel, in my opinion.

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Pavel A.
Sent: Saturday, October 15, 2011 8:25 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] How does MSFT debug mysterious code in Windows?

wrote in message news:xxxxx@ntdev…
> …
> > In my experience, if a driver dev has even HEARD of an Arium, they
> > seem to recall that they’re “very very very expensive”, when in fact
> > that’s no longer the case… they’re merely “very expensive”
>
> Development since then has moved to places where it’s still 3*very, if not
> 4.
> – pa
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

Actually, windbg and kd support being attached to an Arium, or an ITP,
assuming the vendor of that device bothers to write the glue code. We use
that sort of thing occasionally.

To answer the original question, though, it’s more common to just modify the
code so that the debugger doesn’t disconnect in those interstitial times.
The side effect is that it can’t correctly reinitialize at the next step,
but that’s usually okay for the task at hand.

Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group

This post implies no warranties and confers no rights.

wrote in message news:xxxxx@ntdev…

Sadly, I think it would sell about 5 copies.

There aren’t that many of us who have Ariums, I bet. Even fewer are used by
“software” people to debug drivers. Which is too bad, because if more
people DID use these devices there WOULD be better tools for using them the
way we do.

In my experience, if a driver dev has even HEARD of an Arium, they seem to
recall that they’re “very very very expensive”, when in fact that’s no
longer the case… they’re merely “very expensive”

Peter
OSR

Hi Jake,

That’s interesting… We poor man doesn’t have the luxury to modify the code
not to shutdown the debugger. So I could just skip the “disconnect kd” call
from the debugger while it’s still intact? I’ll try that on Monday.

Thanks,
Calvin

On Sun, Oct 16, 2011 at 12:23 PM, Jake Oshins
wrote:

> Actually, windbg and kd support being attached to an Arium, or an ITP,
> assuming the vendor of that device bothers to write the glue code. We use
> that sort of thing occasionally.
>
> To answer the original question, though, it’s more common to just modify
> the code so that the debugger doesn’t disconnect in those interstitial
> times. The side effect is that it can’t correctly reinitialize at the next
> step, but that’s usually okay for the task at hand.
>
> Jake Oshins
> Hyper-V I/O Architect
> Windows Kernel Group
>
> This post implies no warranties and confers no rights.
>
> ------------------------------ ------------------------------ –
>
> wrote in message news:xxxxx@ntdev…
>
>

>
> Sadly, I think it would sell about 5 copies.
>
> There aren’t that many of us who have Ariums, I bet. Even fewer are used
> by “software” people to debug drivers. Which is too bad, because if more
> people DID use these devices there WOULD be better tools for using them the
> way we do.
>
> In my experience, if a driver dev has even HEARD of an Arium, they seem to
> recall that they’re “very very very expensive”, when in fact that’s no
> longer the case… they’re merely “very expensive”
>
> Peter
> OSR
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.**cfm?name=ListServerhttp:
></http:>

Hello!

“In the Windows boot scenario I described, my ROM code on my SAN/Converged NIC adapter needs to fetch boot files from SAN fabric on behalf of bootmgr/winload. I’m not sure if bochs supports such
environment.”

I’ve not tried, but for me Bocks seem to have the ability to:
2) use your own BIOS Image:
http://bochs.sourceforge.net/doc/docbook/user/bochsrc.html#AEN1508

1. map Host’s PCI devices:
   http://bochs.sourceforge.net/doc/docbook/user/bochsrc.html#AEN1839

GP