IMSR not triggered for Win Srv 2012 on ESXi pass-through

NTDEV
1

I have a device driver running for Win Srv 2012 as host without problem. However when using it for Win Srv 2012 running as a VM guest OS on ESXi 6.0 in PCIe pass-through mode, the interrupt does not working anymore. Does anybody has the idea why?

Thanks,

Ying

2

I think you’re going to have bad luck with this question on this forum. Your driver works fine on the native OS? Cool. If it fails to work on a guest of ESXi, it’s likely a configuration problem with ESXi. I happen to think VMware is the devil, but I’ve “had a bad experience”.

I’d recommend checking your interrupt status conditions with the debugger verify interrupts aren’t making their way to your guest environment. It may also depend on what type of PCIe device you have. I’ve run into some interesting issues with PCIe in an FPGA core that presented itself as a PCI bridge.

Good luck, unfortunately for you the right place to get support for this is probably through (a) the less than helpful support community, or (b) the ridiculously expensive paid support from VMware.

3

Hi Shane,

Thanks for your response and valuable suggestions. Is there any chance that ESXi configs guest operating systems differently e.g. Windows different from Linux? The reason I am asking this question is that for the same ESXi, when guest VM is running Linux instead of Windows Server 2012, the message interrupts are pass-through to the guest Linux correctly.

Thanks,

Ying

4

Hi Shane,

One more thing is that after my windows driver is installed in the guest Windows srv 2012 R2, I can see that in the Resources Tab of the Device Properties page from the device manager, there are four interrupt resources displayed as

IRQ 0xFFFFFFD9 (-39)
IRQ 0xFFFFFFD8 (-40)
IRQ 0xFFFFFFD7 (-41)
IRQ 0xFFFFFFD6 (-42)

If there is issue for IMSR pass-through to the guest Windows srv 2012, I should not see the four
IRQs, right?

Thanks,

Ying

5

Ying,

How much debugging have you done in your driver? It looks like ESXi is doing it’s job of passing through the device resources. I would,

  1. Verify that your driver is getting these resources appropriately
  2. Verify that you’re able to get interrupt status updates by reading from your status registers directly

If your interrupt status registers are reflecting your interrupt conditions appropriately, but you aren’t entering your ISR, I think you’re on a good path to solving your issue.

6

Shane,

Thanks for your response and suggestions. I did the some experiments according your two suggestions:

  1. Verified my windows driver was getting the device resources correctly. Basically I managed to disabled the host reset of the card in my windows driver. As the result, the IMSR could be triggered.

  2. Without disabling the host reset, I were able to get interrupt status updates from the status registers directly.

According the design, the host reset of the card has to be there when the driver is loaded. My suspicious is that the host reset zeroizes some of the pcie configure space registers and Windows does not recover the original value. However, my comparison of the pcie configuration space registers (0x00~0xf0)before reset and after the reset does not yield any difference.

Next thing I am going to check is if there is any difference for the MSI-X Vectors before the reset and after the reset.

Do you have any other suggestions where I should look?

Thanks,

Ying

7

xxxxx@yahoo.ca wrote:

According the design, the host reset of the card has to be there when the driver is loaded.

That’s a bad design. It should NEVER be necessary to reset a piece of
hardware during normal operation. Reset should be reserved for
debugging and error recovery. If your hardware collects bad state
during normal operation, then the hardware has a bug.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.