I was wondering if anybody has a NUMA example for a KWDM driver. I’ve
implemented some code and it works fine on a system with one group, but
on a machine with 2 groups, the system hangs.
Where does it hang? In EvtDeviceAdd? Or later on after you return?
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of QuasiCodo
Sent: Wednesday, November 10, 2010 12:01 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] NUMA example
I was wondering if anybody has a NUMA example for a KWDM driver. I’ve implemented some code and it works fine on a system with one group, but on a machine with 2 groups, the system hangs.
Through a series of unfortunate events, I don’t know where the hang
occurred. We (this did not include myself) did not have a debugger
connected to the target system when we were testing this and now I have
to wait in line to get access to the system. Once I do, I will get the
debugger connected and hopefully get some good debug information. But
until then, we are flying blind.
But the fact that you did not way “You can’t do that” is encouraging.
That means at least we have code which looks plausible.
Thx,
((&->
On 11/10/2010 1:09 PM, Doron Holan wrote:
Where does it hang? In EvtDeviceAdd? Or later on after you return?
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of QuasiCodo
Sent: Wednesday, November 10, 2010 12:01 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] NUMA example
I was wondering if anybody has a NUMA example for a KWDM driver. I’ve implemented some code and it works fine on a system with one group, but on a machine with 2 groups, the system hangs.
You use the node number to tweak interrupt settings, so the hang seems to be
caused by mishandling of the interrupt. A typical symptom.
While waiting for the machine, put some debug stuff around your ISR.
–pa
“QuasiCodo” wrote in message news:xxxxx@ntdev… > Through a series of unfortunate events, I don’t know where the hang > occurred. We (this did not include myself) did not have a debugger > connected to the target system when we were testing this and now I have to > wait in line to get access to the system. Once I do, I will get the > debugger connected and hopefully get some good debug information. But > until then, we are flying blind. > > But the fact that you did not way “You can’t do that” is encouraging. That > means at least we have code which looks plausible. > > Thx, > > ((&-> > > > On 11/10/2010 1:09 PM, Doron Holan wrote: >> Where does it hang? In EvtDeviceAdd? Or later on after you return? >> >> -----Original Message----- >> From: xxxxx@lists.osr.com >> [mailto:xxxxx@lists.osr.com] On Behalf Of QuasiCodo >> Sent: Wednesday, November 10, 2010 12:01 PM >> To: Windows System Software Devs Interest List >> Subject: [ntdev] NUMA example >> >> I was wondering if anybody has a NUMA example for a KWDM driver. I’ve >> implemented some code and it works fine on a system with one group, but >> on a machine with 2 groups, the system hangs. >> >> Here is what I am doing in EvtDeviceAdd(): >> >> wdmPdo = WdfDeviceWdmGetPhysicalDevice (device); >> >> status = IoGetDeviceNumaNode (wdmPdo,&nodeNum); >> fdoContext->driverNumaNode = (uint64_t)nodeNum; >> if(NT_SUCCESS(status)) >> { >> memset (&fdoContext->grpAffinity, >> 0, >> sizeof(GROUP_AFFINITY)); >> KeQueryNodeActiveAffinity ( >> fdoContext->driverNumaNode, >> &fdoContext->grpAffinity, >> NULL); >> if (fdoContext->grpAffinity.Mask) >> { >> extPolicy.Policy = WdfIrqPolicySpecifiedProcessors; >> extPolicy.Priority = WdfIrqPriorityHigh; >> extPolicy.TargetProcessorSetAndGroup.Mask = >> fdoContext->grpAffinity.Mask; >> extPolicy.TargetProcessorSetAndGroup.Group = >> fdoContext->driverNumaNode; >> >> WdfInterruptSetExtendedPolicy ( >> fdoContext->wdfInterrupt, >> &extPolicy); >> } >> >> If I remove this code, then everything works fine (except all devices are >> in group 0). Anything wrong with this code? >> >> Thx, >> >> ((&-> >> >> —
I finally got my hands on a 128 processor system, got the debugger
attached via 1394 and then reloaded the driver.
Instead of a bugcheck in the debugger, the system simply rebooted.
Dang! Now I need to come up with someway to debug this. I guess I can
try the checked build of Windows.
((&->
On 11/10/2010 1:01 PM, QuasiCodo wrote:
I was wondering if anybody has a NUMA example for a KWDM driver. I’ve
implemented some code and it works fine on a system with one group, but
on a machine with 2 groups, the system hangs.
Got it working. All I had to do was delete one line:
extPolicy.Priority = WdfIrqPriorityHigh;
Apparently, once you call WDF_INTERRUPT_EXTENDED_POLICY_INIT(), you
should not change anything except the .Policy,
.TargetProcessorSetAndGroup.Group and .TargetProcessorSetAndGroup.Mask
members of the WDF_INTERRUPT_EXTENDED_POLICY structure. Strange.
((&->
On 11/17/2010 11:07 AM, QuasiCodo wrote:
I finally got my hands on a 128 processor system, got the debugger
attached via 1394 and then reloaded the driver.
Instead of a bugcheck in the debugger, the system simply rebooted. Dang!
Now I need to come up with someway to debug this. I guess I can try the
checked build of Windows.
((&->
On 11/10/2010 1:01 PM, QuasiCodo wrote:
> I was wondering if anybody has a NUMA example for a KWDM driver. I’ve
> implemented some code and it works fine on a system with one group, but
> on a machine with 2 groups, the system hangs.
>
> Here is what I am doing in EvtDeviceAdd():
>
> wdmPdo = WdfDeviceWdmGetPhysicalDevice (device);
>
> status = IoGetDeviceNumaNode (wdmPdo, &nodeNum);
> fdoContext->driverNumaNode = (uint64_t)nodeNum;
> if(NT_SUCCESS(status))
> {
> memset (&fdoContext->grpAffinity,
> 0,
> sizeof(GROUP_AFFINITY));
> KeQueryNodeActiveAffinity (
> fdoContext->driverNumaNode,
> &fdoContext->grpAffinity,
> NULL);
> if (fdoContext->grpAffinity.Mask)
> {
> extPolicy.Policy = WdfIrqPolicySpecifiedProcessors;
> extPolicy.Priority = WdfIrqPriorityHigh;
> extPolicy.TargetProcessorSetAndGroup.Mask =
> fdoContext->grpAffinity.Mask;
> extPolicy.TargetProcessorSetAndGroup.Group =
> fdoContext->driverNumaNode;
>
> WdfInterruptSetExtendedPolicy (
> fdoContext->wdfInterrupt,
> &extPolicy);
> }
>
> If I remove this code, then everything works fine (except all devices
> are in group 0). Anything wrong with this code?
>
> Thx,
>
> ((&->
>
It is indeed very strange why removing that line, it works.
but it seems that you have an issue in your code:
extPolicy.TargetProcessorSetAndGroup.Group =
fdoContext->driverNumaNode;
i.e., you are using node# as group#. These are not the same.
you want to do this:
extPolicy.TargetProcessorSetAndGroup = fdoContext->grpAffinity;
This will also set a reserved field in that struct.
Another thing I changed was to use the Group Number instead of the Numa
Node Number. This was a slight misunderstanding with the original code.
((&->
On 11/17/2010 6:17 PM, QuasiCodo wrote:
Got it working. All I had to do was delete one line:
extPolicy.Priority = WdfIrqPriorityHigh;
Apparently, once you call WDF_INTERRUPT_EXTENDED_POLICY_INIT(), you
should not change anything except the .Policy,
.TargetProcessorSetAndGroup.Group and .TargetProcessorSetAndGroup.Mask
members of the WDF_INTERRUPT_EXTENDED_POLICY structure. Strange.
((&->
On 11/17/2010 11:07 AM, QuasiCodo wrote:
> I finally got my hands on a 128 processor system, got the debugger
> attached via 1394 and then reloaded the driver.
>
> Instead of a bugcheck in the debugger, the system simply rebooted. Dang!
> Now I need to come up with someway to debug this. I guess I can try the
> checked build of Windows.
>
> ((&->
>
> On 11/10/2010 1:01 PM, QuasiCodo wrote:
>> I was wondering if anybody has a NUMA example for a KWDM driver. I’ve
>> implemented some code and it works fine on a system with one group, but
>> on a machine with 2 groups, the system hangs.
>>
>> Here is what I am doing in EvtDeviceAdd():
>>
>> wdmPdo = WdfDeviceWdmGetPhysicalDevice (device);
>>
>> status = IoGetDeviceNumaNode (wdmPdo, &nodeNum);
>> fdoContext->driverNumaNode = (uint64_t)nodeNum;
>> if(NT_SUCCESS(status))
>> {
>> memset (&fdoContext->grpAffinity,
>> 0,
>> sizeof(GROUP_AFFINITY));
>> KeQueryNodeActiveAffinity (
>> fdoContext->driverNumaNode,
>> &fdoContext->grpAffinity,
>> NULL);
>> if (fdoContext->grpAffinity.Mask)
>> {
>> extPolicy.Policy = WdfIrqPolicySpecifiedProcessors;
>> extPolicy.Priority = WdfIrqPriorityHigh;
>> extPolicy.TargetProcessorSetAndGroup.Mask =
>> fdoContext->grpAffinity.Mask;
>> extPolicy.TargetProcessorSetAndGroup.Group =
>> fdoContext->driverNumaNode;
>>
>> WdfInterruptSetExtendedPolicy (
>> fdoContext->wdfInterrupt,
>> &extPolicy);
>> }
>>
>> If I remove this code, then everything works fine (except all devices
>> are in group 0). Anything wrong with this code?
>>
>> Thx,
>>
>> ((&->
>>
>
>
Interesting. IIRC, someone from MS recently wrote that interrupt
priorities are not actually implemented?
–pa
“QuasiCodo” wrote in message news:xxxxx@ntdev… > Got it working. All I had to do was delete one line: > > extPolicy.Priority = WdfIrqPriorityHigh; > > Apparently, once you call WDF_INTERRUPT_EXTENDED_POLICY_INIT(), you > should not change anything except the .Policy, > .TargetProcessorSetAndGroup.Group and .TargetProcessorSetAndGroup.Mask > members of the WDF_INTERRUPT_EXTENDED_POLICY structure. Strange. > > ((&-> > > On 11/17/2010 11:07 AM, QuasiCodo wrote: >> I finally got my hands on a 128 processor system, got the debugger >> attached via 1394 and then reloaded the driver. >> >> Instead of a bugcheck in the debugger, the system simply rebooted. Dang! >> Now I need to come up with someway to debug this. I guess I can try the >> checked build of Windows. >> >> ((&-> >> >> On 11/10/2010 1:01 PM, QuasiCodo wrote: >>> I was wondering if anybody has a NUMA example for a KWDM driver. I’ve >>> implemented some code and it works fine on a system with one group, but >>> on a machine with 2 groups, the system hangs. >>> >>> Here is what I am doing in EvtDeviceAdd(): >>> >>> wdmPdo = WdfDeviceWdmGetPhysicalDevice (device); >>> >>> status = IoGetDeviceNumaNode (wdmPdo, &nodeNum); >>> fdoContext->driverNumaNode = (uint64_t)nodeNum; >>> if(NT_SUCCESS(status)) >>> { >>> memset (&fdoContext->grpAffinity, >>> 0, >>> sizeof(GROUP_AFFINITY)); >>> KeQueryNodeActiveAffinity ( >>> fdoContext->driverNumaNode, >>> &fdoContext->grpAffinity, >>> NULL); >>> if (fdoContext->grpAffinity.Mask) >>> { >>> extPolicy.Policy = WdfIrqPolicySpecifiedProcessors; >>> extPolicy.Priority = WdfIrqPriorityHigh; >>> extPolicy.TargetProcessorSetAndGroup.Mask = >>> fdoContext->grpAffinity.Mask; >>> extPolicy.TargetProcessorSetAndGroup.Group = >>> fdoContext->driverNumaNode; >>> >>> WdfInterruptSetExtendedPolicy ( >>> fdoContext->wdfInterrupt, >>> &extPolicy); >>> } >>> >>> If I remove this code, then everything works fine (except all devices >>> are in group 0). Anything wrong with this code? >>> >>> Thx, >>> >>> ((&-> >>> >> >> > >
That’s true. They were in early Vista builds. They didn’t make any
shipping product.
Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group
This post implies no warranties and confers no rights.
“Pavel A.” wrote in message news:xxxxx@ntdev…
Interesting. IIRC, someone from MS recently wrote that interrupt
priorities are not actually implemented?
–pa
“QuasiCodo” wrote in message news:xxxxx@ntdev… > Got it working. All I had to do was delete one line: > > extPolicy.Priority = WdfIrqPriorityHigh; > > Apparently, once you call WDF_INTERRUPT_EXTENDED_POLICY_INIT(), you should > not change anything except the .Policy, .TargetProcessorSetAndGroup.Group > and .TargetProcessorSetAndGroup.Mask members of the > WDF_INTERRUPT_EXTENDED_POLICY structure. Strange. > > ((&-> > > On 11/17/2010 11:07 AM, QuasiCodo wrote: >> I finally got my hands on a 128 processor system, got the debugger >> attached via 1394 and then reloaded the driver. >> >> Instead of a bugcheck in the debugger, the system simply rebooted. Dang! >> Now I need to come up with someway to debug this. I guess I can try the >> checked build of Windows. >> >> ((&-> >> >> On 11/10/2010 1:01 PM, QuasiCodo wrote: >>> I was wondering if anybody has a NUMA example for a KWDM driver. I’ve >>> implemented some code and it works fine on a system with one group, but >>> on a machine with 2 groups, the system hangs. >>> >>> Here is what I am doing in EvtDeviceAdd(): >>> >>> wdmPdo = WdfDeviceWdmGetPhysicalDevice (device); >>> >>> status = IoGetDeviceNumaNode (wdmPdo, &nodeNum); >>> fdoContext->driverNumaNode = (uint64_t)nodeNum; >>> if(NT_SUCCESS(status)) >>> { >>> memset (&fdoContext->grpAffinity, >>> 0, >>> sizeof(GROUP_AFFINITY)); >>> KeQueryNodeActiveAffinity ( >>> fdoContext->driverNumaNode, >>> &fdoContext->grpAffinity, >>> NULL); >>> if (fdoContext->grpAffinity.Mask) >>> { >>> extPolicy.Policy = WdfIrqPolicySpecifiedProcessors; >>> extPolicy.Priority = WdfIrqPriorityHigh; >>> extPolicy.TargetProcessorSetAndGroup.Mask = >>> fdoContext->grpAffinity.Mask; >>> extPolicy.TargetProcessorSetAndGroup.Group = >>> fdoContext->driverNumaNode; >>> >>> WdfInterruptSetExtendedPolicy ( >>> fdoContext->wdfInterrupt, >>> &extPolicy); >>> } >>> >>> If I remove this code, then everything works fine (except all devices >>> are in group 0). Anything wrong with this code? >>> >>> Thx, >>> >>> ((&-> >>> >> >> > >
“Jake Oshins” wrote in message news:xxxxx@ntdev… > That’s true. They were in early Vista builds. They didn’t make any > shipping product. > > Jake Oshins > Hyper-V I/O Architect > Windows Kernel Group
Then why WdfIrqPriorityHigh matters?
– pa
> -------------------------------------------------------------- > “Pavel A.” wrote in message news:xxxxx@ntdev… > > Interesting. IIRC, someone from MS recently wrote that interrupt > priorities are not actually implemented? > --pa > > “QuasiCodo” wrote in message news:xxxxx@ntdev… >> Got it working. All I had to do was delete one line: >> >> extPolicy.Priority = WdfIrqPriorityHigh; >> >> Apparently, once you call WDF_INTERRUPT_EXTENDED_POLICY_INIT(), you >> should not change anything except the .Policy, >> .TargetProcessorSetAndGroup.Group and .TargetProcessorSetAndGroup.Mask >> members of the WDF_INTERRUPT_EXTENDED_POLICY structure. Strange. >> >> ((&-> >> >> On 11/17/2010 11:07 AM, QuasiCodo wrote: >>> I finally got my hands on a 128 processor system, got the debugger >>> attached via 1394 and then reloaded the driver. >>> >>> Instead of a bugcheck in the debugger, the system simply rebooted. Dang! >>> Now I need to come up with someway to debug this. I guess I can try the >>> checked build of Windows. >>> >>> ((&-> >>> >>> On 11/10/2010 1:01 PM, QuasiCodo wrote: >>>> I was wondering if anybody has a NUMA example for a KWDM driver. I’ve >>>> implemented some code and it works fine on a system with one group, but >>>> on a machine with 2 groups, the system hangs. >>>> >>>> Here is what I am doing in EvtDeviceAdd(): >>>> >>>> wdmPdo = WdfDeviceWdmGetPhysicalDevice (device); >>>> >>>> status = IoGetDeviceNumaNode (wdmPdo, &nodeNum); >>>> fdoContext->driverNumaNode = (uint64_t)nodeNum; >>>> if(NT_SUCCESS(status)) >>>> { >>>> memset (&fdoContext->grpAffinity, >>>> 0, >>>> sizeof(GROUP_AFFINITY)); >>>> KeQueryNodeActiveAffinity ( >>>> fdoContext->driverNumaNode, >>>> &fdoContext->grpAffinity, >>>> NULL); >>>> if (fdoContext->grpAffinity.Mask) >>>> { >>>> extPolicy.Policy = WdfIrqPolicySpecifiedProcessors; >>>> extPolicy.Priority = WdfIrqPriorityHigh; >>>> extPolicy.TargetProcessorSetAndGroup.Mask = >>>> fdoContext->grpAffinity.Mask; >>>> extPolicy.TargetProcessorSetAndGroup.Group = >>>> fdoContext->driverNumaNode; >>>> >>>> WdfInterruptSetExtendedPolicy ( >>>> fdoContext->wdfInterrupt, >>>> &extPolicy); >>>> } >>>> >>>> If I remove this code, then everything works fine (except all devices >>>> are in group 0). Anything wrong with this code? >>>> >>>> Thx, >>>> >>>> ((&-> >>>> >>> >>> >> >> >
No idea. I doubt that that was the problem in his code. As far as I can
tell, the value is completely ignored.
Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group
This post implies no warranties and confers no rights.
“Pavel A.” wrote in message news:xxxxx@ntdev…
“Jake Oshins” wrote in message news:xxxxx@ntdev… > That’s true. They were in early Vista builds. They didn’t make any > shipping product. > > Jake Oshins > Hyper-V I/O Architect > Windows Kernel Group
Then why WdfIrqPriorityHigh matters?
– pa
> -------------------------------------------------------------- > “Pavel A.” wrote in message news:xxxxx@ntdev… > > Interesting. IIRC, someone from MS recently wrote that interrupt > priorities are not actually implemented? > --pa > > “QuasiCodo” wrote in message news:xxxxx@ntdev… >> Got it working. All I had to do was delete one line: >> >> extPolicy.Priority = WdfIrqPriorityHigh; >> >> Apparently, once you call WDF_INTERRUPT_EXTENDED_POLICY_INIT(), you >> should not change anything except the .Policy, >> .TargetProcessorSetAndGroup.Group and .TargetProcessorSetAndGroup.Mask >> members of the WDF_INTERRUPT_EXTENDED_POLICY structure. Strange. >> >> ((&-> >> >> On 11/17/2010 11:07 AM, QuasiCodo wrote: >>> I finally got my hands on a 128 processor system, got the debugger >>> attached via 1394 and then reloaded the driver. >>> >>> Instead of a bugcheck in the debugger, the system simply rebooted. Dang! >>> Now I need to come up with someway to debug this. I guess I can try the >>> checked build of Windows. >>> >>> ((&-> >>> >>> On 11/10/2010 1:01 PM, QuasiCodo wrote: >>>> I was wondering if anybody has a NUMA example for a KWDM driver. I’ve >>>> implemented some code and it works fine on a system with one group, but >>>> on a machine with 2 groups, the system hangs. >>>> >>>> Here is what I am doing in EvtDeviceAdd(): >>>> >>>> wdmPdo = WdfDeviceWdmGetPhysicalDevice (device); >>>> >>>> status = IoGetDeviceNumaNode (wdmPdo, &nodeNum); >>>> fdoContext->driverNumaNode = (uint64_t)nodeNum; >>>> if(NT_SUCCESS(status)) >>>> { >>>> memset (&fdoContext->grpAffinity, >>>> 0, >>>> sizeof(GROUP_AFFINITY)); >>>> KeQueryNodeActiveAffinity ( >>>> fdoContext->driverNumaNode, >>>> &fdoContext->grpAffinity, >>>> NULL); >>>> if (fdoContext->grpAffinity.Mask) >>>> { >>>> extPolicy.Policy = WdfIrqPolicySpecifiedProcessors; >>>> extPolicy.Priority = WdfIrqPriorityHigh; >>>> extPolicy.TargetProcessorSetAndGroup.Mask = >>>> fdoContext->grpAffinity.Mask; >>>> extPolicy.TargetProcessorSetAndGroup.Group = >>>> fdoContext->driverNumaNode; >>>> >>>> WdfInterruptSetExtendedPolicy ( >>>> fdoContext->wdfInterrupt, >>>> &extPolicy); >>>> } >>>> >>>> If I remove this code, then everything works fine (except all devices >>>> are in group 0). Anything wrong with this code? >>>> >>>> Thx, >>>> >>>> ((&-> >>>> >>> >>> >> >> >