Interrupt latency on debugee machine

While testing a synchronous communications driver on build 2195 free I
noticed that I was missing some interrupts on the debugee machine. I was
running 115,200 bps and was not outputting any debug messages. Furthur
investigation with a scope showed that when running the debugger the
interrupt latency for my interrupt was approx 50 millisecs. Without the
debugger running the latency was approx 10 - 20 microseconds. It would
appear that a tremendous amount of data is running across the debug port ?
I never missed interrupts on NT 4. Maybe I have something set up wrong ?