Possible to kernel debug over the internet?

Sven_Kreamer · January 24, 2015, 5:33am

I am unable to debug a machine in another country. Below is the output from the kernel debugger (ip addresses omitted). Any suggestions on if this might be resolvable?

–

Using NET for debugging
Opened WinSock 2.0
Waiting to reconnect…
Connected to target x.x.x.x on port 50000 on local IP y.y.y.y.
Connected to Windows 8 9600 x64 target at (…)), ptr64 TRUE
Kernel Debugger connection established.

************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred
SRV*c:\dev\symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is:
SRV*c:\dev\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Received an out of order encrypted packet. Packet dropped.
Bad packet sent from x.x.x.x.

OSR_Community_User · January 24, 2015, 1:18pm

How frequently are you getting packets in out of order?

The normal pattern for debugger traffic is to send pairs of packets - an ack of the previous packet immediately followed by a data packet containing the requested data. Which is typically then immediately acked by the other side followed by another command packet. So you see 2 packets in sequence from each side, with the smaller one typically being the ACK. This is easy to watch using netmon or wireshark.

If the pair of packets come in swapped order, it usually should not break the debugger, because if the debugger gets an expected data packet without the preceding ack, it treats the ack as received, and processes the data packet. The following ACK is now out of order, and gets dropped by the encryption layer, but was already implicitly processed by the debugger.

So getting occasional out of order packets should not prevent the debugger from working.

If something in the pipe between you and the target is swapping the order on every set of packets sent, that could be a problem. We did not explicitly test that scenario.

A fix for a scenario like that will likely not be resolvable without a change to the debugger, which means you won’t see it for some time.

Furthermore, unless we can repro the problem here, I am not going to make any changes to KDNET to try to handle this. We have successfully used KDNET to debug machines on the other side of the world just fine.

It was part of what we tested when KDNET was first developed.

Now granted, that was on Microsoft’s internal corporate network, but we debugged machines in India from Redmond, without issue.

The first thing I would check is if something in your infrastructure (ie: NIC driver, or router/switch) is unnecessarily switching the order of packets.

The encryption layer in KDNET does not allow packets received out of order to be accepted. In order to prevent replay attacks.

Given that changing that behavior, raises potential security issues, makes me much less inclined to make changes that would enable acceptance of out of order packets.

Joe.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@gmail.com
Sent: Saturday, January 24, 2015 2:34 AM
To: Kernel Debugging Interest List
Subject: [windbg] Possible to kernel debug over the internet?

I am unable to debug a machine in another country. Below is the output from the kernel debugger (ip addresses omitted). Any suggestions on if this might be resolvable?

–

Using NET for debugging
Opened WinSock 2.0
Waiting to reconnect…
Connected to target x.x.x.x on port 50000 on local IP y.y.y.y.
Connected to Windows 8 9600 x64 target at (…)), ptr64 TRUE Kernel Debugger connection established.

************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred
SRV*c:\dev\symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is:
SRV*c:\dev\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Received an out of order encrypted packet. Packet dropped.
Bad packet sent from x.x.x.x.

WINDBG is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Sven_Kreamer · January 24, 2015, 7:50pm

> How frequently are you getting packets in out of order?

Out of order packet errors happen immediately after the connection is made. Sometimes there are numerous ones listed up to about 10 instead of just one. And from there on nothing happens. It doesn’t sound like the problem is well understood or that a solution is forthcoming, but thanks for the analysis.

Pavel_A1 · January 25, 2015, 9:50am

On 25-Jan-2015 02:51, xxxxx@gmail.com wrote:

> How frequently are you getting packets in out of order?

Out of order packet errors happen immediately after the connection is made. Sometimes there are numerous ones listed up to about 10 instead of just one. And from there on nothing happens. It doesn’t sound like the problem is well understood or that a solution is forthcoming, but thanks for the analysis.

Do you use some IP-over-IP or Ethernet-over-IP tunnel, or just connect
“normally”?
– pa

OSR_Community_User · January 26, 2015, 1:43pm

Try setting the debugger to break at boot.

Ctrl-alt-k in windbg.

If you get an initial connection, and then after hitting ‘g’ you get into a state during early boot where nothing seems to be happening, hit ctrl-break in windbg, and see if you can break in. If so, then just hit ‘g’ again.

You might have to rinse and repeat a couple of times if your connection is truly terrible.

Sometimes on bad connections the debugger and target get in a state where the debugger things the target is running, while the target thinks it is broken in, and both just sit quietly doing nothing, but the target doesn’t run because it is broken in waiting for the debugger to tell it what to do. Since the debugger thinks the target is running, it sends nothing at all to the target. The trivial work around is to hit ctrl-break in the debugger, at which point the two should get back in sync (since the debugger will now think the target is broken in, which it is) and you can tell the target to continue execution, or run debugger commands, etc.

Joe.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Pavel A.
Sent: Sunday, January 25, 2015 6:50 AM
To: Kernel Debugging Interest List
Subject: Re:[windbg] Possible to kernel debug over the internet?

On 25-Jan-2015 02:51, xxxxx@gmail.com wrote:

> How frequently are you getting packets in out of order?

Out of order packet errors happen immediately after the connection is made. Sometimes there are numerous ones listed up to about 10 instead of just one. And from there on nothing happens. It doesn’t sound like the problem is well understood or that a solution is forthcoming, but thanks for the analysis.

Do you use some IP-over-IP or Ethernet-over-IP tunnel, or just connect “normally”?
– pa

WINDBG is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Sven_Kreamer · February 1, 2015, 9:25pm

Breaking in didn’t work. Switched to openVPN and received a completely different error shown below. I am sure the keys are identical. Tried the auto generated key and a manual key to be sure.

Packet failed authentication. Please make sure the host and target encryption
keys match exactly. Also make sure you don’t have multiple target machines
pointed at the same network port on your host.
Bad packet sent from x.x.x.x.
Run nslookup x.x.x.x from a command prompt to get the machine name.

Tim_Roberts · February 2, 2015, 1:13pm

xxxxx@gmail.com wrote:

Breaking in didn’t work. Switched to openVPN and received a completely different error shown below. I am sure the keys are identical. Tried the auto generated key and a manual key to be sure.

Packet failed authentication. Please make sure the host and target encryption
keys match exactly. Also make sure you don’t have multiple target machines
pointed at the same network port on your host.
Bad packet sent from x.x.x.x.
Run nslookup x.x.x.x from a command prompt to get the machine name.

It’s not obvious to me that Ethernet debugging is supposed to work
through one or more NAT gateways. Has that ever been promised?

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.