Faster kernel debugging?

Hello,

I’m having the following issue: I’m finding kernel debugging very slow (comparing to normal userspace apps debugging).

I have tried the following methods:

  1. Debug a VM
    – Good sides:
    revert to snapshot, no longer need to uninstall the driver, in case the OS seems to have gotten unstable for reason (or I messed up something in there), undo to initial state.

– Bad sides: The only way it seems to work is via serial port / named pipes, but this (or the fact that is VM is the cause?) causes step-by-step debugging to be very slow (like, it can happen a “step over” to last ~5 seconds), the debugger seems more likely to yield timeout errors, code not to break at breakpoints, etc.

I have tried with a VM on SSD disk, but there’s not much difference. Booting seems quick enough, but that’s quite it. Step-by-step debugging is still killing.

  1. Debug a non-virtual machine (tried with my laptop)
    – Good sides:
    step-by-step debugging is quite instantaneous (no need to wait), the debugger commands are quick and the debugger & debuggee answer quickly. I connected the PC to the laptop via ethernet cable.

– Bad sides:
No “revert to snapshot”: shutdown may last ~6 seconds, any modification of the driver code requires uninstalling the driver, crashes at driver uninstall are killing: I must restart and enter safe mode to uninstall, then restart, to be able to install a new version. I find restarts a time killer even if the laptop’s disk is SSD (but it does work better than with an HDD). I’m debugging Hyper-V Server 2012, and the time it waits most at shutdown is shutting down VMMS.

So the question is: is there any way to speed up more the debugging & deployment?
The best solution thus far I found to be debugging the laptop, which has SSD as disk.

Thanks!

Actually, the shutting down of VMMS lasts about 30 seconds. That is evil.

Have you tried VirtualKD ?

Regards,

Julian

El domingo, 29 de diciembre de 2013, escribi?:

Actually, the shutting down of VMMS lasts about 30 seconds. That is evil.


WINDBG is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Like Julian said. Go VirtualKD.

Personally, I use my own tool for debugging Windows XP to Windows 7, and switched to KD over network if the target is Windows 8 or newer.

Also to speed things up, you can try to reduce number of open windows (call stack, local variables, memory windows), also you can prefix variables in watch windows with YOURDRIVER!VarName.

L.

>I’m having the following issue: I’m finding kernel debugging very slow

(comparing to normal userspace apps debugging).

Use a good 1394 connection (like the LSI chip based cards) between the debugger machine and target. This kind of connection can single step at about keyboard repeat rate (20 steps/sec), and can take a full memory dump (or other operation like !irpfind that scans memory) at almost 40 Mbytes/sec. LSI based PCIe 1394 cards are about $50 each (two needed) and a 1394b cable for $15.

The advantage of the LSI chipset based 1394 cards is they can do direct memory read/write above the 4 GB line totally in hardware, and the wire runs at 800 Mbps. Older 1394 cards do 400 Mbps on the wire, and for transfers at higher addresses have to invoke code on the target to marshal the data.

I know some people still like legacy serial for kernel debugging, because it works correctly though power state transitions. I personally have found power state debugging is like 0.1% of my debugging, so would rather use the fastest possible connection the other 99.9% of the time.

It also helps to have systems that reboot fast. I have some Supermicro server motherboards that can reboot in 30 seconds. I’ve also worked on servers from a couple of companies that take like 5 minutes+ to reboot. These servers are perhaps optimal for rebooting once a year in your data center, but are pretty painful to do driver development on. It also generally degrades reboot time to have a lot of ram. 4Gb or 8GB is usually plenty on the target machine, unless you’re getting into things like server performance optimization.

Debugging a VM guest is only useful if you’re debugging a driver that has no hardware. Being able snapshot a VM is handy to get back to the correct debugging context, but a 1394b debugging connection is still rather higher performance.

On Win 8 and later you can also debug over some Ethernet cards, which is slower than 1394 but way faster than serial. Ethernet debugging goes at about 1-2 MBytes/sec, which is still pretty slow if you want to write a dump file. Ethernet debugging can also have seriously degraded performance if your network has congestion or things have to go through a deep inspection firewall.

Jan

I should try VirtualKD. I’ll have to get a new SSD for that.

What I’m debugging now is not a device driver, so a VM can be considered as an option as well. And the serial port / named pipe seems to be the only option for a VM (I tried with networking, the virtualized NIC was one registered in the msdn doc regarding NICs, but that didn’t work).

Regarding debugging the laptop: I don’t have a 1394 port in the laptop, and I’m guessing such a card cannot be put in a laptop. Anyway, I’m pretty satisfied with the speed via ethernet cable, for the operations I’ve done thus far. I’m having a direct-cable connection, and two NICs in the PC: one for internet, the other for direct-cable connection. And the target OS has firewall off.

My problem with rebooting is that it lasts 30+ seconds normally, about 1+ minutes when you need to uninstall the driver via safe mode (because it crashes at uninstall). And any assertion failure / crash means you need to reboot. And I don’t know if it’s only me, but I’m having lots of these (i.e. when I’m writing new code, until I make it do stuff correctly). In normal userspace apps, a trivial problem such as a wrong condition specified for an assertion can be fixed + the app re-run in debug mode, is solved within a few seconds, yet any trivial mistake like this costs you a lot of time in kernel.

On Mon, Dec 30, 2013 at 6:33 PM, Jan Bottorff wrote:
>>I’m having the following issue: I’m finding kernel debugging very slow
>> (comparing to normal userspace apps debugging).
>
> Use a good 1394 connection (like the LSI chip based cards) between the debugger machine and target. This kind of connection can single step at about keyboard repeat rate (20 steps/sec), and can take a full memory dump (or other operation like !irpfind that scans memory) at almost 40 Mbytes/sec. LSI based PCIe 1394 cards are about $50 each (two needed) and a 1394b cable for $15.
>
> The advantage of the LSI chipset based 1394 cards is they can do direct memory read/write above the 4 GB line totally in hardware, and the wire runs at 800 Mbps. Older 1394 cards do 400 Mbps on the wire, and for transfers at higher addresses have to invoke code on the target to marshal the data.
Are you referring to the cards with LSI FW643E chipset? It would great
to know which specific one actually works with KD.

Thanks
Kris

>
> I know some people still like legacy serial for kernel debugging, because it works correctly though power state transitions. I personally have found power state debugging is like 0.1% of my debugging, so would rather use the fastest possible connection the other 99.9% of the time.
>
> It also helps to have systems that reboot fast. I have some Supermicro server motherboards that can reboot in 30 seconds. I’ve also worked on servers from a couple of companies that take like 5 minutes+ to reboot. These servers are perhaps optimal for rebooting once a year in your data center, but are pretty painful to do driver development on. It also generally degrades reboot time to have a lot of ram. 4Gb or 8GB is usually plenty on the target machine, unless you’re getting into things like server performance optimization.
>
> Debugging a VM guest is only useful if you’re debugging a driver that has no hardware. Being able snapshot a VM is handy to get back to the correct debugging context, but a 1394b debugging connection is still rather higher performance.
>
> On Win 8 and later you can also debug over some Ethernet cards, which is slower than 1394 but way faster than serial. Ethernet debugging goes at about 1-2 MBytes/sec, which is still pretty slow if you want to write a dump file. Ethernet debugging can also have seriously degraded performance if your network has congestion or things have to go through a deep inspection firewall.
>
> Jan
>
> —
> WINDBG is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


Kris

If you search the archives of this list, about a year ago I posted a detailed message about a specific card I was using that worked great. I also just ordered a $30 Apple Thunderbolt to 1394 adapter, which a teardown reported uses a LSI chipset, and will report back here in a few weeks if it works for Windows kernel debugging. The boards I’ve been using have an LSI FW643-w chip. I believe it’s described online at http://www.unibrain.com/products/fireboard800-e-1394b-pci-express-adapter/

Jan

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Krzysztof Uchronski
Sent: Friday, January 3, 2014 10:16 AM
To: Kernel Debugging Interest List
Subject: Re: [windbg] Faster kernel debugging?

On Mon, Dec 30, 2013 at 6:33 PM, Jan Bottorff wrote:
>>I’m having the following issue: I’m finding kernel debugging very slow
>>(comparing to normal userspace apps debugging).
>
> Use a good 1394 connection (like the LSI chip based cards) between the debugger machine and target. This kind of connection can single step at about keyboard repeat rate (20 steps/sec), and can take a full memory dump (or other operation like !irpfind that scans memory) at almost 40 Mbytes/sec. LSI based PCIe 1394 cards are about $50 each (two needed) and a 1394b cable for $15.
>
> The advantage of the LSI chipset based 1394 cards is they can do direct memory read/write above the 4 GB line totally in hardware, and the wire runs at 800 Mbps. Older 1394 cards do 400 Mbps on the wire, and for transfers at higher addresses have to invoke code on the target to marshal the data.
Are you referring to the cards with LSI FW643E chipset? It would great to know which specific one actually works with KD.

Thanks
Kris

>
> I know some people still like legacy serial for kernel debugging, because it works correctly though power state transitions. I personally have found power state debugging is like 0.1% of my debugging, so would rather use the fastest possible connection the other 99.9% of the time.
>
> It also helps to have systems that reboot fast. I have some Supermicro server motherboards that can reboot in 30 seconds. I’ve also worked on servers from a couple of companies that take like 5 minutes+ to reboot. These servers are perhaps optimal for rebooting once a year in your data center, but are pretty painful to do driver development on. It also generally degrades reboot time to have a lot of ram. 4Gb or 8GB is usually plenty on the target machine, unless you’re getting into things like server performance optimization.
>
> Debugging a VM guest is only useful if you’re debugging a driver that has no hardware. Being able snapshot a VM is handy to get back to the correct debugging context, but a 1394b debugging connection is still rather higher performance.
>
> On Win 8 and later you can also debug over some Ethernet cards, which is slower than 1394 but way faster than serial. Ethernet debugging goes at about 1-2 MBytes/sec, which is still pretty slow if you want to write a dump file. Ethernet debugging can also have seriously degraded performance if your network has congestion or things have to go through a deep inspection firewall.
>
> Jan
>
> —
> WINDBG is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer


Kris


WINDBG is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Found it (“Anybody know which brand of 1394 card supports > 4GB
addressing”). Thanks Jan.

Kris

.

On Sat, Jan 4, 2014 at 12:02 AM, Jan Bottorff wrote:
> If you search the archives of this list, about a year ago I posted a detailed message about a specific card I was using that worked great. I also just ordered a $30 Apple Thunderbolt to 1394 adapter, which a teardown reported uses a LSI chipset, and will report back here in a few weeks if it works for Windows kernel debugging. The boards I’ve been using have an LSI FW643-w chip. I believe it’s described online at http://www.unibrain.com/products/fireboard800-e-1394b-pci-express-adapter/
>
> Jan
>
> -----Original Message-----
> From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Krzysztof Uchronski
> Sent: Friday, January 3, 2014 10:16 AM
> To: Kernel Debugging Interest List
> Subject: Re: [windbg] Faster kernel debugging?
>
> On Mon, Dec 30, 2013 at 6:33 PM, Jan Bottorff wrote:
>>>I’m having the following issue: I’m finding kernel debugging very slow
>>>(comparing to normal userspace apps debugging).
>>
>> Use a good 1394 connection (like the LSI chip based cards) between the debugger machine and target. This kind of connection can single step at about keyboard repeat rate (20 steps/sec), and can take a full memory dump (or other operation like !irpfind that scans memory) at almost 40 Mbytes/sec. LSI based PCIe 1394 cards are about $50 each (two needed) and a 1394b cable for $15.
>>
>> The advantage of the LSI chipset based 1394 cards is they can do direct memory read/write above the 4 GB line totally in hardware, and the wire runs at 800 Mbps. Older 1394 cards do 400 Mbps on the wire, and for transfers at higher addresses have to invoke code on the target to marshal the data.
> Are you referring to the cards with LSI FW643E chipset? It would great to know which specific one actually works with KD.
>
> Thanks
> Kris
>
>>
>> I know some people still like legacy serial for kernel debugging, because it works correctly though power state transitions. I personally have found power state debugging is like 0.1% of my debugging, so would rather use the fastest possible connection the other 99.9% of the time.
>>
>> It also helps to have systems that reboot fast. I have some Supermicro server motherboards that can reboot in 30 seconds. I’ve also worked on servers from a couple of companies that take like 5 minutes+ to reboot. These servers are perhaps optimal for rebooting once a year in your data center, but are pretty painful to do driver development on. It also generally degrades reboot time to have a lot of ram. 4Gb or 8GB is usually plenty on the target machine, unless you’re getting into things like server performance optimization.
>>
>> Debugging a VM guest is only useful if you’re debugging a driver that has no hardware. Being able snapshot a VM is handy to get back to the correct debugging context, but a 1394b debugging connection is still rather higher performance.
>>
>> On Win 8 and later you can also debug over some Ethernet cards, which is slower than 1394 but way faster than serial. Ethernet debugging goes at about 1-2 MBytes/sec, which is still pretty slow if you want to write a dump file. Ethernet debugging can also have seriously degraded performance if your network has congestion or things have to go through a deep inspection firewall.
>>
>> Jan
>>
>> —
>> WINDBG is sponsored by OSR
>>
>> OSR is hiring!! Info at http://www.osr.com/careers
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>
>
>
> –
> Kris
>
> —
> WINDBG is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>
> —
> WINDBG is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


Kris