CreateFileW fails with AccessDenied unless I have a second device instance

I’m using DeviceIoControl to send IOCTL commands to my IddCx-based (virtual) video adapter driver. If I install the driver and create a single instance of the virtual display adapter…:

DEVCON.EXE INSTALL my-virtual-display-driver.inf MyVirtualDisplayModel

When I run my IOCTL test program (As Administrator), I get Access Denied when I call CreateFileW to get a handle to the adapter…:

// Open the virtual display device
hDevice = ::CreateFileW(
    devicePath.c_str(),                 // Name of the device to be opened
    GENERIC_WRITE | GENERIC_READ,       // Requested access to the device
    FILE_SHARE_READ | FILE_SHARE_WRITE, // Requested sharing mode of the device
    nullptr,                            // Default security attributes
    OPEN_EXISTING,                      // Disposition
    0,                                  // Device attributes and flags
    nullptr);                           // No template file to copy file attributes from

…but completely by accident, I discovered that if I manually create another instance of my virtual display adapter (in Device Manager via Action > Add legacy hardware), the CreateFileW call succeeds.

Can anyone shed light on this? Is the OS somehow holding an exclusive handle to the first instance?

OK, turns out this was PEBKAC – There was a DLL doing some stuff on load and unload that I wasn’t aware of… resulting in two CreateFile calls within the same process. So it was the sharing…

Ugh. Spoke too soon. :#

I’m back to getting Access Denied, without having changed the code, so it’s something environmental?

Just to be sure, I looped through all the possible combinations of DesiredAccess, SharingMode, and FlagsAndAttributes (e.g. FILE_ATTRIBUTE_NORMAL) for the CreateFile function, and I get Access Denied for every possible combination.

I install my driver using DEVCON, and it creates a single instance (same as before). Checking the single device instance’s properties in Device Manager, Device instance path is ROOT\DISPLAY\0000.

My driver defines a custom device interface:

// An interface GUID ({580FE7DF-137A-45F1-9998-7B1F006CDC43}) so that apps can find and communicate with this device
DEFINE_GUID(GUID_DEVINTERFACE_IMVD, 0x580fe7df, 0x137a, 0x45f1, 0x99, 0x98, 0x7b, 0x1f, 0x0, 0x6c, 0xdc, 0x43);

...

// Create a device interface for this virtual display adapter device
ntStatus = ::WdfDeviceCreateDeviceInterface(hWdfAdapterDevice, &Public::GUID_DEVINTERFACE_IMVD, nullptr);

I’m using CM_Get_Device_Interface_ListW to get the device path, passing in the interface GUID and nullptr for the pDeviceID argument. I get back a single string \\?\ROOT#DISPLAY#0000#{580fe7df-137a-45f1-9998-7b1f006cdc43}.

The intent is to get a handle to the Display Adapter device itself (not a specific monitor), and then use IOCTL commands to enable (i.e. plug in) and configure monitors. The driver no longer creates a single (plugged in) monitor by default when an adapter instance is created. Could that be the issue?

Is there some way (using SysInternals, say) to look deeper into why I’m getting AccessDenied back from CreateFile?

Using SysInternals’ Handle Viewer (Handle64.exe) to search for the device, I do see that the System process (pid: 4) seems to have a handle open:

C:\Users\Scott\Desktop\SysInternals Suite>handle -nobanner -a -u 580fe7

System             pid: 4      type: SymbolicLink  \<unable to open process> 2C44: \GLOBAL??\ROOT#DISPLAY#0000#{580fe7df-137a-45f1-9998-7b1f006cdc43}

Not sure whether this is significant or not…