How to find an available interrupt vector that doesn't conflict.

I’ve written a driver to access the Instruction-Based Sampling (or IBS) hardware on newer AMD CPUs.
I need to select an interrupt vector, which I can link to my ISR and also tell the IBS hdwe which vector to generate its interrupt on. The hdwe doesn’t have a predefined vector, although it does have a predefined LVT entry in the local APIC.
Being a newbie to windows drivers, and not wanting to take the time to master the art completely, I came up with a method. I have a four-level loop where I try IoConnectInterrupt with various parameter values. In order, the loop tries:
* exclusive, shared access
* latched, level sensitive
* vectors 50 through FE
* IRQLs 0 through 9.
On my system, the first success is with exclusive, latched, vect 80h, IRQL 0. I’ve also tried testing shared access first, then the result is shared, with the other values the same.
Unfortunately, after enabling IBS interrupts and servicing several of them, the system completely hangs. No mouse, no keyboard, no console update for my client that is monitoring progress.

This code used to work a few years ago, when I had Windows 7 and an earlier AMD processor. Now I’ve got Windows 10 and an AMD Ryzen and I get the hangups. I figure I was just plain lucky that it used to work before.

Anyway, I’m sure there’s a RIGHT way to do this, but I don’t know what that would be. I’m asking for help from you all, at least pointers to stuff I can read.

I’ve got a few inklings…

  1. I need to use the PnP manager. Even though the IBS hdwe is not an external device, I have the impression that PnP makes sure everything works in cooperation.

  2. Maybe the DIRQL of 0 is wrong, and should be higher?

  3. There’s something in the Hal layer that manages interrupt vectors, which I can use.

  4. Reserving an interrupt vector for the IBS hdwe in the Registry somehow.

Another bit of info: AMD has a free program called CodeXL that takes IBS samples. Its kernel drivers operate without system hangups, so obviously it’s possible. I’m going through disassembly of the driver code to figure out what it’s doing to get its ISR called. So far, I’ve seen a call to HalAllocateHardwareCounters, which will prevent other (well-behaved) drivers from accessing the IBS hardware. Perhaps this is what I need to do in my own IBS driver too?

Here is the list of relevant imports for the two CodeXL drivers, if that may help you discern what their method is. Curious that IoConnectInterrupt(Ex) is not among them. Do you recognize which of these functions will bind an ISR to an interrups?

DLL Name: ntoskrnl.exe
ExFreePoolWithTag
KeInitializeEvent
KeWaitForSingleObject
DbgPrint
IoAllocateWorkItem
ExRegisterCallback
ObfDereferenceObject
ExCreateCallback
RtlInitUnicodeString
IoQueueWorkItemEx
IoFreeWorkItem
RtlIsNtDdiVersionAvailable
HalDispatchTable
ExUnregisterCallback
KeGetCurrentProcessorNumberEx
IoDeleteDevice
IoDeleteSymbolicLink
IoCreateDevice
ExAllocatePoolWithTag
IoCreateSymbolicLink
KeQueryActiveProcessorCountEx
PsGetCurrentProcessId
PsGetCurrentThreadId
MmIsAddressValid
KeAcquireSpinLockRaiseToDpc
KeRemoveQueueDpc
ExSetTimerResolution
KeSetTimerEx
KeInitializeTimer
KeReleaseSpinLock
KeCancelTimer
KeClearEvent
KeSetEvent
KeInitializeDpc
KeInsertQueueDpc
KeGetProcessorNumberFromIndex
KeSetTargetProcessorDpcEx
MmMapIoSpace
MmUnmapIoSpace
IofCompleteRequest
KeSetImportanceDpc
__C_specific_handler

DLL Name: HAL.dll
KeQueryPerformanceCounter
HalAllocateHardwareCounters
HalFreeHardwareCounters
HalGetBusDataByOffset

DLL Name: ntoskrnl.exe
ObReferenceObjectByHandle
ExAllocatePoolWithTag
KeReleaseGuardedMutex
ExFreePoolWithTag
KeAcquireGuardedMutex
ExEventObjectType
DbgPrint
KeGetCurrentProcessorNumberEx
MmUnlockPages
PsRemoveLoadImageNotifyRoutine
ZwOpenSection
ZwUnmapViewOfSection
MmProbeAndLockPages
PsSetLoadImageNotifyRoutine
IoAllocateMdl
RtlInitUnicodeString
PsRemoveCreateThreadNotifyRoutine
ZwMapViewOfSection
IoFreeMdl
MmMapLockedPagesSpecifyCache
PsSetCreateThreadNotifyRoutine
PsSetCreateProcessNotifyRoutine
ZwClose
IoDeleteDevice
IoDeleteSymbolicLink
IoCreateDevice
IofCompleteRequest
IoCreateSymbolicLink
KeQueryActiveProcessorCountEx
ZwCreateFile
ZwWriteFile
ZwQueryInformationFile
KeSetEvent
ExInterlockedInsertTailList
ExInterlockedRemoveHeadList
PsIsThreadTerminating
PsCreateSystemThread
KeClearEvent
KeWaitForSingleObject
PsTerminateSystemThread
KeSetImportanceDpc
KeInsertQueueDpc
KeInitializeDpc
RtlQueryRegistryValues
KeInitializeEvent
IoSizeofWorkItem
ZwQueryVolumeInformationFile
IoQueryFileDosDeviceName
IoInitializeWorkItem
IoQueueWorkItemEx
ObfReferenceObject
IoUninitializeWorkItem
ZwOpenFile
IoIs32bitProcess
ObfDereferenceObject
KeInitializeGuardedMutex
IoGetStackLimits
RtlVirtualUnwind
MmSystemRangeStart
RtlQueryModuleInformation
MmHighestUserAddress
ProbeForRead
ExpInterlockedFlushSList
KeInitializeApc
ExpInterlockedPopEntrySList
ExpInterlockedPushEntrySList
PsGetCurrentProcessId
KeInsertQueueApc
PsGetThreadId
PsGetThreadProcessId
InitializeSListHead
RtlImageDirectoryEntryToData
RtlIsNtDdiVersionAvailable
KeGetProcessorNumberFromIndex
KeSetTargetProcessorDpcEx
PsGetCurrentProcessWow64Process
MmIsAddressValid
__C_specific_handler

DLL Name: HAL.dll
HalRequestSoftwareInterrupt
KeQueryPerformanceCounter

Why haven’t you tried hitting your test computer with a hammer(or at least writing zero to CR0) just to check if it may be of any help? Look - in terms of chances of success your “methodology” does not seem to fare any better than the one that I have suggested…

BTW, a combination of IRQL 0 with interrupt vector 0xFE seems to be particularly impressive …

Anton Bassov

Taking a quick look, this thing doesn’t appear to be interrupt driven but
like most CPU features uses MSRs. Why don’t you start out with the Linux
sample that’s easy to find ?
https://github.com/jlgreathouse/AMD_IBS_Toolkit/blob/master/README.txt

//Daniel

xxxxx@rolle.name wrote:

I’ve written a driver to access the Instruction-Based Sampling (or IBS) hardware on newer AMD CPUs.

This code used to work a few years ago, when I had Windows 7 and an earlier AMD processor. Now I’ve got Windows 10 and an AMD Ryzen and I get the hangups. I figure I was just plain lucky that it used to work before.

Anyway, I’m sure there’s a RIGHT way to do this, but I don’t know what that would be.

The RIGHT way to do this is to have an entry in the BIOS ACPI DSDT for
this interrupt.  Then, Windows will create an interrupt with the
parameters already established, and will pass the resource to whatever
driver gets loaded for that ACPI device.

Why does this process need an interrupt?

  1. Maybe the DIRQL of 0 is wrong, and should be higher?

Absolutely, it is wrong.  IRQL of 0 is PASSIVE_LEVEL  DIRQLs are
typically equal to the vector number (although that could have been an
architectural accident on the machines where I was looking).

Here is the list of relevant imports for the two CodeXL drivers, if that may help you discern what their method is. Curious that IoConnectInterrupt(Ex) is not among them. Do you recognize which of these functions will bind an ISR to an interrups?

Does that driver have an INF file?  Is it matching an ACPI device code? 
Or is it a legacy driver that’s attaching to the interrupt through
hackery?  The presence of HalRequestSoftwareInterrupt suggests hackery,
although AMD probably participated in writing the HAL and understands
its proper use.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Daniel, thanks and I’ll check out the link you gave me. But if it’s for Linux, it won’t be much help, as I already know how to use the MSRs for the IBS.

Tim, I don’t know how CodeXL’s drivers are installed. When I install CodeXL, the drivers appear in System32/drivers and in the SCM’s database.

I suspect that the call to HalRequestSoftwareInterrupt is not what registers the ISR. I couldn’t find any documentation for this routine. From the disassembly, it appears that it is being called with a single argument of 1 in the CL register. Anyway, this call is in one of the drivers, and the actual ISR, as far as I can tell, is in the other driver and it’s address is not exported.

As I mentioned before, do you recognize any functions in the FIRST group that will hook an interrupt vector?

I did find in the AMD driver where it sets the vector number in the LVT entry. The number is 1. The rest of the bits in the LVT are set to 0.

Does that make sense to you, that interrupt 1 would be dedicated to the IBS?

I’m still looking to find the place where the vector is hooked to the ISR.

AMD’s IBS uses interrupts because it has a cycling counter, and when the counter rolls over, it signals an interrupt (if they are enabled). The ISR can then read information from various MSRs and reenable interrupts and sampling for the next counter rollover. This way, the driver can capture periodic samples of whatever is running on the core where the ISR is hooked to the interrupt.

Now I could have my driver just read the MSRs on a control request from the client, and return the sample values if they exist, and reenable sampling for the next rollover. All this without using interrupts. If I can’t get interrupts to work right, I’ll have to resort to this and have my client app use the driver to poll the IBS. It won’t be as good for my purposes, but it will do if need be.

Regarding DIRQL = 0… I wasn’t sure if IoConnectInterrupt would succeed for some values and not for others, so I decided to try a whole range. Now maybe all I need to do is pick a level and just use that. Is DISPATCH_LEVEL the lowest level that is valid for an ISR? Or does it need to be higher?

Regarding the interrupt vector… If AMD and the BIOS are doing the “right” thing, it may be that there’s something in the BIOS that determines the interrupt vector. I know that the hardware uses a dedicated APIC LVT entry, whose index is given in an MSR. I assumed I had to choose a vector and then store it in the LVT entry. But perhaps the BIOS and/or the system has selected a vector and stored it there already.

I’ll look at the APIC LVT initial state before I try to change it and see if there’s something meaningful there. If not, can you tell me how to explore the BIOS ACPI DSDT from my app, or from a kernel driver, to see if there’s anything there regarding the IBS. Can you give me a reference explaining what this DSDT is and its format?

> I suspect that the call to HalRequestSoftwareInterrupt is not what registers the ISR.

Indeed…

I couldn’t find any documentation for this routine.

This routine requests an interrupt on the CPU (either a self-interrupt or IPI, depending in the argument) by writing to the local APIC’s ICR. It is not supposed to be used by driver writers,and this is the reason why it is not documented…

Regarding DIRQL = 0… I wasn’t sure if IoConnectInterrupt would succeed for some values
and not for others, so I decided to try a whole range. Now maybe all I need to do is
pick a level and just use that.

What yo really need to do is to learn what the very concept of IRQL is all about, and then discover how it is actually implemented. At this point you will realize how stupid all your"experimentation" is. Basically, this is just an index into the array that stores vector numbers - the higher the index is, the higher the numerical value of the vector is. Interrupt priority is implied by the vector number on x86 and x86_64 systems

do you recognize any functions in the FIRST group that will hook an interrupt vector?

The only functions that hook an ISR are IoConnectInterrupt()and IoConnectInterruptEx().

However, your are supposed to call these functions only with the right parameters, i.e. the ones that you get from resource descriptors. What they do is registering your ISR with a call chain that gets invoked by interrupt handler stub (i.e. the one that the address specified in IDT points to) when an interrupt that corresponds to a given vector number occurs. It has absolutely nothing to do with mapping IRQ lines to vectors or writing a handler address to IDT. This part gets done at the boot time, based upon the information that the OS gets from BIOS (i.e. concerning the relationship between the PCI/ PCI-X devices and interrupt lines and pins) .

In order to realize how stupid your "experimentation"is, consider the scenario when you physically share an interrupt line and pin with another PCI device(s), which is specified in the BIOS settings, but still want to get a separate interrupt vector and provide your very own arguments to IoConnectInterrupt()…

Anton Bassov

xxxxx@rolle.name wrote:

Tim, I don’t know how CodeXL’s drivers are installed. When I install CodeXL, the drivers appear in System32/drivers and in the SCM’s database.

Regarding the interrupt vector… If AMD and the BIOS are doing the “right” thing, it may be that there’s something in the BIOS that determines the interrupt vector. I know that the hardware uses a dedicated APIC LVT entry, whose index is given in an MSR. I assumed I had to choose a vector and then store it in the LVT entry. But perhaps the BIOS and/or the system has selected a vector and stored it there already.

Perhaps.  Have you actually asked AMD about this?  I assume they have
the same kind of detailed support forums that Intel has, and those
forums are usually pretty active.  I’m sure there are AMD nuts on their
forums who would be happy to tell you way more than you want to know
about this.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Thanks, Tim and Anton.

The short story is that, apparently, the CodeXL driver does not use interrupts from the IBS hardware at all, hence no interrupt vector is involved. Rather, it uses a timer interrupt to periodically poll the IBS to see if a sample is available.

Regarding help from AMD community, there is a DevGurus forum, but there’s very little activity on it. They have a CodeXL subforum which hasn’t had any activity since 4 months ago. The general discussions subforum is more active. I’m going to join the forum and post my question and see. Also, there’s a general email for AMD support that I’ll try. If I learn anything I’ll post it here.

I was mistaken about using vector number 1. I thought that the value was being stored in a word that would later go into the LVT, but I had the wrong base register in the disassembly. So I still don’t know where the driver is getting the vector number.

Since there is no call to IoConnectInterrupt(Ex), I am going to assume the driver is using a repeating timer interrupt. In fact, the function that checks the IBS hardware for a rollover and reads the info MSRs and reenables the sampling is called Deferred_TimerInterrupt
Service, and looks like it is scheduled by KeInitializeTimer.

I’m guessing also that where the driver sets the IBS’ LVT entry, it is doing so to disable the interrupt, rather than enable it and store a vector. The IBS hardware always signals an interrupt when a sample is taken, and the LVT determines whether the processor is actually interrupted.