Initializing the serial ports for transferring data

NTDEV
1

I’m trying to use the serial port (COM) to send/receive data over the serial device in order to make a new kernel debugger for Windows.

Actually, it’s not possible to use NT APIs for this purpose, it’s because I want to halt the system by raising the IRQL and masking the interrupts and also want to use the serial port in polling mode.

I use the following code to initialize the serial port.

VOID CpInitialize( PCPPORT Port, PUCHAR  Address, ULONG  Rate)
{
PUCHAR hwport;
UCHAR   mcr, ier;

Port->Address = Address;
Port->Baud = 0;

CpSetBaud(Port, Rate);

// Assert DTR, RTS.
hwport = Port->Address;
hwport += COM_MCR;

mcr = MC_DTRRTS;
_outp(hwport, mcr);

hwport = Port->Address;
hwport += COM_IEN;

ier = 0;
_outp(hwport, ier);
}

and pass 115200 as Rate and COM2_PORT as the Address.

#define COM1_PORT   0x03f8
#define COM2_PORT   0x02f8
#define COM3_PORT   0x03e8
#define COM4_PORT   0x02e8

And finally, for sending a byte out, I use the following function.

VOID CpPutByte(PCPPORT  Port, 	UCHAR   Byte)
{
if (KdDbgPortsPresent == FALSE) {
	return;
}
//  Wait for port to not be busy
while (!(CpReadLsr(Port, COM_OUTRDY) & COM_OUTRDY));

// Send the byte
_outp(Port->Address + COM_DAT, Byte);
}

In order to test the above functions, I create a Serial Device in VMWare Workstation and connect to the Windbg debugger using kdnet. (I don’t use COM for debugging of Windbg.).

The problem is nothing transfers to the other side but if use I a usermode applications then I can successfully transfer data over the serial but it’s not possible to send bytes over serial using the above code.

I further investigated the problem and find a sample from Microsoft in SDK which is called “kdserial”.

After using the “kdserial” codes, I have the exact same problem and tracing into the code, I find that here is the guilty code :

// Check to see if all bits are set in LSR. If this is the case, it means
// the port I/O address is invalid as 0xFF is nonsense for LSR. This
// prevents writing a byte to non-existent hardware.

Lsr = Port->Read(Port, COM_LSR);
if (Lsr == SERIAL_LSR_NOT_PRESENT) {
return UartNotReady;
}

It seems that the port is not initialized as the LSR shows “SERIAL_LSR_NOT_PRESENT=0xff”.
.
.
So, my questions are :
1. According to what I read, Serial ports are I/O mapped and not Memory Mapped I/O (MMIO). Am I right?
2. The second question is, what’s wrong with my code that makes LSR, 0xff, and I can’t send data over this serial port? Did I miss anything?

2

I’m trying to use the serial port (COM) to send/receive data over the serial device in order to make a new kernel debugger for Windows.

Why would you waste your time with this? The kernel debugger is deeply embedded in Windows, has more than 30 years of development and testing experience, and benefits from knowledge of the internal implementations that you can never have. What could you possibly gain by embarking on such an adventure?

  1. According to what I read, Serial ports are I/O mapped and not Memory Mapped I/O (MMIO). Am I right?

Correct. At least, physical serial ports are. You aren’t dealing with a physical serial port. You’re dealing with VMWare’s virtualized simulation of a serial port. Does it react like a physical port? Who knows? Does it do the setup early enough in the boot process to allow a kernel debugger? Who knows? We know that serial debugging in a VM works, but only if one maps the VM’s serial port to a named pipe.

Have you tried this on a bare metal system yet?

3

Why would you waste your time with this? The kernel debugger is deeply embedded in Windows, has more than 30 years of development and testing experience, and benefits from knowledge of the internal implementations that you can never have. What could you possibly gain by embarking on such an adventure?
Thousands of reasons, my debugger is a Hypervisor-assisted debug.
Here are some reasons,

Does Windbg support advanced Extended Page Table (EPT) techniques for debugging? No,
Is Windbg able to simulate hardware read/write access to a page (not 4 debugger registers I mean a PAGE)? No,
Does Windbg use Intel PT to get coverage? No,
Does Windbg support Hidden hooks with splitting VPIDs? No,
Does Windbg support special mechanisms to make itself hidden? No,
… and many many features that my debugger supports them but Windbg doesn’t have them, of course, my debugger would never be like Windbg but sure it has its own special features that make it unique.

Have you tried this on a bare-metal system yet?

No, unfortunately, don’t have any computer that supports a physical serial port, and don’t know how to test it. (If you have any idea for testing it, pls let me know, e.g buy some peripherals or blah blah).

I also, reversed the kdcom.dll and it seems that it has the same initialization part but when I write it on my debugger then it has the above-mentioned problem.

4

You have enough time to do all this work, but can’t find a system with a physical serial port? Most systems at least have the header as the circuitry only costs a few cents. add in serial cards star at around $10. buy a premium one fo $50

https://www.amazon.com/s?k=serial+card&ref=nb_sb_noss_2

apparently you can still buy cards with parallel connectors! I didn’t see any with the anti-static shields or spring locks we used to have, but the fact that they still exist is remarkable.

Anyways, this is a project that will likely require thousands of hours of work on your part, and ultimately be unmaintainable since you will have no control over what changes Microsoft will choose to make in the future. As an education exercise, it would be excellent I think; but as a commercial venture, not so much. Please feel free to pursue this. No matter what I think about the merits of the project, I will all try to help you if I can