system freezing when testing the driver on multiple ports

Hi All,

On my sysetm i have 36 searil ports. I wrote a driver for driving an individual serial port and in installed this driver on these ports. My driver creates a separate device object for each instance of the serial port my driver.

I am facing the below problem.

While running the read write tests in on multiple serial ports simultaneously, the tests are running smoothly for a while and then the system is freezing. (system stops responding).
When i run the test on a single port or till 4 ports, the test runs smoothly. However when i increase the number of ports to 8 the test runs for a while and the system freezes.

Below are my observations.

  1. I checked the CPU Usage and memory at the time of running the tests. CPU usage is remains within 10% all the time and memory usage stays within 600 MB (even at the time of freeze). So the possibilities of CPU overload or running out of memory can be ruled out.
  2. Suspecting a deadlock I enabled driver verifier (verifier.exe and enabled all the checks). I expected a crash in the event of a deadlock or a memory leak. I didn?t see a crash. So probably there is no dead lock or a memory leak in the driver

My suspicion is that the driver is running out of some resource when I run multiple instances of the driver on multiple ports. I initially suspected memory usage because I was creating 20K memory for each device instance but then since memory usage is not increasing when I run the tests on multiple ports, I ruled out this possibility.

Brief about my test application:

The test application that I use basically performs overlapped I/O on serial ports. It creates two threads for writing to the port and reading from the same port. (I have my ports connected in a loop back fashion and hence the port receives the same data which is written to it).The Write thread calls WriteFIle function and the read thread calls ReadFile function on the serial port. These functions are called in a for loop for a user specified number of iterations.

WriteFIle and ReadFile functions are called in an asynchronous fashion (using Overlapped structure). We run an instance of this test application for each serial port. (I have a total of 36 serial ports on my system and this driver is expected run smoothly with all the 36 ports simultaneously.)

I am trying to understand, what other resource apart from CPU and memory could be effecting the run of the driver. SHould i look at any resource that is used at the driver object level? (since it is common for all the device instances)? Could somebody please give me a hint as to how to approach this problem?

PS:

I have been posting a lot of queries my driver offlate. I received some suggestions about changing my driver design. However since i am in short of time for a design overhaul, i couldn’t take it up. So gentlemen who think that this issue could be related to my design may please ignore the query. (hopefully i will still find some hints :slight_smile:

I’m assuming this is a PCI or PCIe based device. Are you using legacy interrupts, MSI or MSI-x? How much time are you spending in your ISRs/DPCs? When you say you have 10% CPU burden for 36 serial ports, this immediately raised my suspicion. Are you viewing kernel time also? Unless you’ve implemented DMA in your UART design I’m guessing what you are seeing is a pegged CPU scrambling to handle interrupts.

Using Perfmon may help you identify any processor usage concerns.

Shane,

You are right…Mine is a PCI-e based device. I am using legacy interrupts.

By Kernel time do you mean how much percentage of CPU time went in running mode? how to actually view this?

I am spending less than 10 micro seconds in my ISR. and somewhere between 50 to 100 micro seconds in the DPC.