I've been single-stepping with WinDbg (new debugger from the store) a kernel memory allocation function (that I believe was called from multiple places, and not just from my code.)
I started single-stepping it (F10), and also (F11 for step-into) and noticed that current core numbers were changing. Say, I started from:
6: kd> p
then it switched to:
8: kd> p
then came back to core 6.
I understand that it was probably catching other cores executing at the same place. But that was super confusing. I wonder if it was done by design, or if it's a sort of a bug in the kernel debugger?
Why do you think it's a bug? That CPU doesn't stay frozen in suspended animation while you paused in the debugger. The thread is frozen, but the CPU is still working. It's quite possible that your process changed CPUs while you were waiting.
I'm guessing you set a breakpoint and then step from that breakpoint?
The debugger behavior you describe is exactly what happens when the first core hits a breakpoint, and then you step, and a different core then hits the breakpoint. The fix is after the first core hits the breakpoint, disable that breakpoint, so you only have a single core than hit the breakpoint and is being stepped. You unfortunately can't directly set a breakpoint for a specific core, even though the cpu usually is capable of doing that with a hardware breakpoint.
If I want to breakpoint on a specific core I often insert a little chunk of debug code that checks the core number and if it matches do some line of code that I can set a breakpoint on that does nothing useful (like increments a global integer variable when the core number matches). You can then set the breakpoint on the code that does that increment, or you can also set a memory write breakpoint on that global as only the desired core will write to it. These are all strategies to avoid multiple cores stopping at a breakpoint. When you are stepping, you can also just say go if the core number was not the same is first hit the breakpoint, which works if it's uncommon to hit the breakpoint. You can also set breakpoints with a condition, with the the condition being the core number matches the one you want. This works but it degrades performance if lots of cores are hitting the breakpoint and then having to run debugger script code to determine if you continue or halt.
Windbg is not able to stop one core, and let all the other ones keep running. When you break into the debugger, all cores will be halted or resumed. JTAG debuggers sometimes can halt/continue individual cores, so you can step through your driver code while the drivers keep executing and responding to interrupts. Some devices get unhappy if their driver is not responding to heartbeat interrupts (like happens if you stop in windbg). They may do things like decide their internal firmware has crashed so will restart their firmware. For cases like this it may may be better to log events, like with TraceLogging, instead of trying to halt/step in the debugger.
Or sometimes it's better to write code that detects when a failure has happened, and trigger a bugcheck, and you can then do post-mortem debugging live or from a crashdump.
When I write a bunch of new kernel code I often like to single step through it when I first bring it up, checking variables have plausible values as I step. Setting the system to a single core with bcdedit sometimes makes this initial live walkthrough easier.
Thanks a lot, @Jan_Bottorff. Yes, I hear you. In my own code I could definitely "assist" with a breakpoint by adding my own instructions around it, like catching the right condition, etc.
In this case though, I'm deep in the weeds of the OS memory manager, so I can't just add stuff to it willy-nilly. I am just trying to step through it to see why I'm getting the result I was not expecting (besides the point of this question.)
And yes indeed, in this situation I set up a conditional breakpoint first (because obviously that specific part of the memory manager gets called A LOT) and then immediately remove that breakpoint. But there are two observations that happen after that:
Even if I expressly remove my initial breakpoint, it still gets hit once or twice afterwards. I am not very clear why. (I'm not sure if this is something specific for an ARM64 CPU, or not? Because that's the h/w architecture that I'm debugging it with.)
I then try to single-step through machine instructions, but quite often my single-step "hits" on multiple cores. So it almost looks like I'm debugging two (or more) threads at the same time, which becomes mighty confusing. And yes, I learned to hit Go to let one thread continue executing. But that is also confusing because I may let go the thread that I was single-stepping. This obviously can be checked, but it just complicates things, which are already quite complex.
I'm not sure if I'm describing it properly. Have you witnessed that as well?
I currently work for an ARM64 server cpu company, so know a lot about debugging on those.
It sounds like the breakpoint is hit by more than 1 core before it's able to freeze all the cores. When it breaks for the first time, do stack dumps on the other cores and see if any also have the breakpoint in their stacks (try !running -t, there are also some use debugger extensions in the MEX package https://www.microsoft.com/en-us/download/details.aspx?id=53304 that can dump the unique stacks of all cores). I know it's kind of a pain, but try just noticing which core number first hits the breakpoint, disable that breakpoint, and then step. If it comes back on a different core, hit go. If it comes back on the same core as the first time, hit step. Stepping ONLY effects the current core, so if other cores break it means they are still processing the initial breakpoint. If there are other cores that also hit the breakpoint, this will get all the ones that didn't hit the breakpoint first back to a running instead of stepping state. Eventually, ONLY the first core to hit the breakpoint will be stepping. If it's a LOT of cores, a debugger macro would help. I work on systems with a few hundred cores, and much of the time it's only a small number of cores that have this secondary breakpoint hit, so manually continuing the extra cores that hit the breakpoint is not much work.
@Jan_Bottorff , thanks. That sounds exactly like what is happening. I wonder if that is a bug or a feature. Or if MSFT didn't care to change it since the KD part of the OS kernel is very old and everyone is afraid of touching it.
Also how do you check if the bp hit on other cores? Do you look for an IPI on that core with the nt!KiFreezeCurrentProcessor?
PS. I sent you a DM so that not to spam this thread. We work in a very similar business.
Idea is to turn the breakpoint location into an infinite loop so if any other threads/cores hit it they'll stall. Also helps to cut the system down to a single CPU.
Thanks, @Scott_Noone_OSR . Arm64 also has a branch to itself instruction. I use it too. Except that in today's OS one needs to be very careful with that approach. The OS can bugcheck if an IPI or a clock interrupt gets missed for too long, which may happen if such a deadloop happens at higher IRQL.
If you need to single step code at elevated IRQL just boot with one CPU. Then you don’t have to worry about the other processors getting in the way of your nature study.