Problem with serial port access under MS Virtual Server

NTDEV
1

Hello,

I have a very simple driver that is used to communicate some information,
during the booting of a VM, back to the host machine. A service runs on the
host machine and monitors the named pipe which is mapped to the COM port in
the VM setup. This code works perfectly under VMWare, but does not work at
all under MS Virtual Server. Specifically, when the driver writes to the
port, it gets a status success, but no data is ever received on the host
machine’s named pipe. However, once the machine boots up, writing bytes to
the same port with hyperterm produces data on the host end.

Here is an overview of what I’m doing in the driver, am I missing some
step(s) to properly initialize and/or write to the port?

  1. IoGetDeviceObjectPointer() on “\device\serial0”, this succeeds and for
    all subsequent operations I use device referenced by the returned
    fileObject->deviceObject

  2. send IOCTL_SERIAL_INTERNAL_BASIC_SETTINGS to device, this succeeds

  3. send IOCTL_SERIAL_SET_TIMEOUTS to device, this succeeds (set a 30 second
    timeout on reads)

  4. call IoBuildSynchronousFsdRequest() to build an IRP_MJ_WRITE on 5 bytes
    of data, this succeeds.

  5. call IoCallDriver() to send the IRP, this succeeds. Specifically,
    IoCallDriver() first returns STATUS_PENDING, then a short while later sets
    the event and IO_STATUS_BLOCK.Status is STATUS_SUCCESS and
    IO_STATUS_BLOCK.Information is set to 5. So it definitely would appear that
    this data got sent “somewhere”.

At this point, if running on VMWare, my service listening on the named pipe
would see the 5 bytes of data, but this same service doesn’t see anything
when using Virtual Server. Is there a buffer or something that could be
queueing up these bytes? I know at the time of these operations that
Virtual Server has already created the named pipe because my service on the
host is able to open it. But somehow it’s acting like the two are not
linked.

Any comments would be greatly appreciated!

Thanks,
Jeff

2

(1) You may want to try it without the 2) send
IOCTL_SERIAL_INTERNAL_BASIC_SETTINGS

Sincerely;
Mike Jones

“Jeff Bromberger” wrote in message
news:xxxxx@ntdev…
> Hello,
>
> I have a very simple driver that is used to communicate some information,
> during the booting of a VM, back to the host machine. A service runs on
> the host machine and monitors the named pipe which is mapped to the COM
> port in the VM setup. This code works perfectly under VMWare, but does
> not work at all under MS Virtual Server. Specifically, when the driver
> writes to the port, it gets a status success, but no data is ever received
> on the host machine’s named pipe. However, once the machine boots up,
> writing bytes to the same port with hyperterm produces data on the host
> end.
>
> Here is an overview of what I’m doing in the driver, am I missing some
> step(s) to properly initialize and/or write to the port?
>
> 1) IoGetDeviceObjectPointer() on “\device\serial0”, this succeeds and for
> all subsequent operations I use device referenced by the returned
> fileObject->deviceObject
>
> 2) send IOCTL_SERIAL_INTERNAL_BASIC_SETTINGS to device, this succeeds
>
> 3) send IOCTL_SERIAL_SET_TIMEOUTS to device, this succeeds (set a 30
> second timeout on reads)
>
> 4) call IoBuildSynchronousFsdRequest() to build an IRP_MJ_WRITE on 5 bytes
> of data, this succeeds.
>
> 5) call IoCallDriver() to send the IRP, this succeeds. Specifically,
> IoCallDriver() first returns STATUS_PENDING, then a short while later sets
> the event and IO_STATUS_BLOCK.Status is STATUS_SUCCESS and
> IO_STATUS_BLOCK.Information is set to 5. So it definitely would appear
> that this data got sent “somewhere”.
>
> At this point, if running on VMWare, my service listening on the named
> pipe would see the 5 bytes of data, but this same service doesn’t see
> anything when using Virtual Server. Is there a buffer or something that
> could be queueing up these bytes? I know at the time of these operations
> that Virtual Server has already created the named pipe because my service
> on the host is able to open it. But somehow it’s acting like the two are
> not linked.
>
> Any comments would be greatly appreciated!
>
> Thanks,
> Jeff
>
>