STORport hang while attempting to use MSI-X

Robert_Randall · March 18, 2011, 3:43pm

It was suggested that I pose my problem to the list and so here it is…

I am working on a STORport miniport driver for an adapter that is
MSI-X capable.

To start with I must confess that I have seen some strange results
from the StorPortInitializePerOpts API call (PERF_CONFIGURATION_DATA
query). ?I was getting DIFFERENT results back during my testing.
Sometimes the NUMA information would be populate, sometimes not,
sometimes ALL the Flags would be set, sometimes just a few. ?Last week
(Thursday) the results of the call became consistent and now I ALWAYS
get the same answer (see below, just 2 flags set). ?I did NOT change
the drivers on the host during this testing time (using Intel’s
drivers on Asus Z8PE-D12 MB). From the PCIe bus trace things are just
fine.

Now, I’m quite confused by the fact that StorPortInitializePerfOpts
was returning inconsistent answers for a while and by the fact that
it is NOT populating EVERYTHING in PERF_CONFIGURATION_DATA since the
MB I’m running on is a NUMA architecture and has PCIe GEN 2 support.

This experience has led me to not trust the hardware (opinions?). ?I
have a new MB on order (SuperMicro), however, here is the latest
information regarding my problem including debug output context
information and the stack trace of the “stuck” STORport call.

f FFFFFA8003CEA680 Initialize 304 Checking for MSI support using
StorPortGetMSIInfo
f FFFFFA8003CEA680 Initialize 319 found 256 MSI-X interrupts available
Break instruction exception - code 80000003 (first chance)
Initialize+0x2fd:
fffff880053189ed cc int 3 // caused by KdBreakPoint() 15: kd> g f FFFFFA8003CEA680 Initialize 368 PerfOpts flags available STOR_PERF_DPC_REDIRECTION yes STOR_PERF_CONCURRENT_CHANNELS yes STOR_PERF_INTERRUPT_MESSAGE_RANGES no STOR_PERF_OPTIMIZE_FOR_COMPLETION_DURING_STARTIO no STOR_PERF_ADV_CONFIG_LOCALITY no f FFFFFA8003CEA680 Initialize 393 requesting use of 3 MSI-X interrupts f FFFFFA8003CEA680 Initialize 434 StorPortInitializePerfOpts Flags driver is USING STOR_PERF_DPC_REDIRECTION yes STOR_PERF_CONCURRENT_CHANNELS no STOR_PERF_INTERRUPT_MESSAGE_RANGES no STOR_PERF_OPTIMIZE_FOR_COMPLETION_DURING_STARTIO no STOR_PERF_ADV_CONFIG_LOCALITY no f FFFFFA8003CEA680 Initialize 435 Driver is using MSI-X MessageNumbers 0 through 2 f FFFFFA8003CEA680 Initialize 436 StorPortInitializePerfOpts was successful f FFFFFA8003CEA680 Initialize 444 exit PassiveInitialize runs - omitted. PassiveInitialize DOES configure the PCIe device resulting in the device generating several MSI-X interrupts which are serviced by the correct ISR. Note: I'm NOT using any DPCs yet (on purpose). The PassiveInitialize thread and the first MsiInterrupt thread DO overlap. I'm running with InterruptSynchronizeAll just to get things working so there is no call to StorPort(Acquire|Release)MSISpinLock in the ISR. Interrupts get serviced. 0 FFFFF80002056CC0 MsiInterrupt 1157 entry MessageID 0 0 FFFFF80002056CC0 MsiInterrupt 1207 exit 0 FFFFF80002056CC0 MsiInterrupt 1157 entry MessageID 0 0 FFFFF80002056CC0 MsiInterrupt 1207 exit After this my miniport driver stack is stuck. The system is not hung or stuck, just my driver stack. The system is idle expect for one CPU. The stack trace of the non idle CPU is: Child-SP RetAddr : Args to Child : Call Site fffff8800274e5b0 fffff88005339e65 : fffffa800803c600
0000000000000000 fffffa8007e10100 fffffa80000f0000 : storport!StorPortPause+0x1aa0 fffff8800274e640 fffff8800537d6a8 : fffffa800803c600
fffffa8007e19010 fffffa8003e45c60 fffff88005326a01 : storport!StorPortGetUncachedExtension+0x11d5 fffff8800274e6a0 fffff8800537d9f0 : fffffa8007e19010
0000000000000001 fffffa800803c600 fffffa8007e19010 : storport!StorPortInitialize+0x3de18 fffff8800274e6f0 fffff8800537db94 : fffffa8007e19010
fffff88005343110 fffffa800803c4b0 fffff8800274e820 : storport!StorPortInitialize+0x3e160 fffff8800274e750 fffff8000228d17e : fffffa8007e19010
fffffa8008031750 fffffa800803c4b0 fffff88002113180 : storport!StorPortInitialize+0x3e304 fffff8800274e790 fffff80001fc3b2d : fffffa8003fc38c0
fffffa8008031750 fffff80001fcd250 0000000000000000 : nt!EtwUnregister+0x33e fffff8800274e7d0 fffff8000229c4c6 : fffff800020d2200
fffffa8003fc4010 fffffa8008031750 fffffa8003fc41b8 : nt!RtlAssert+0x33d fffff8800274e890 fffff8000229c764 : fffffa8003fc4010
fffffa8003fc003e fffffa8003fc4010 0000000000000001 : nt!IoGetDeviceInterfaces+0x2ac6 fffff8800274e920 fffff800022bfe96 : fffffa8003fc4010
fffffa8003fc4010 0000000000000000 0000000000000000 : nt!IoGetDeviceInterfaces+0x2d64 fffff8800274e950 fffff800022c0287 : fffffa8003fc4010
0000000000000000 0000000000000001 fffff80002141c38 : nt!ExCreateCallback+0x3ef6 fffff8800274ebc0 fffff80001fcfb83 : 0000000100000003
0000000000000000 0000000000000001 0000000000000000 : nt!ExCreateCallback+0x42e7 fffff8800274ec10 fffff80001ee0a21 : fffff80001fcf870
fffff80001e1ea01 fffff80002073600 fffffa8003cea680 : nt!DbgSetDebugFilterState+0x2343 fffff8800274ecb0 fffff80002173cce : 0000000000000000
fffffa8003cea680 0000000000000080 fffffa8003cd51f0 : nt!KeAcquireInStackQueuedSpinLockAtDpcLevel+0x2f1 fffff8800274ed40 fffff80001ec7fe6 : fffff880022d3180
fffffa8003cea680 fffff880022de2c0 0000000000000000 : nt!PsCreateSystemThread+0x1da fffff8800274ed80 0000000000000000 : fffff8800274f000
fffff88002749000 fffff8800274d8d0 00000000`00000000 :
nt!KeInitializeSemaphore+0x24a

Any thoughts from you smart folk out there in driver land?

Best,
Robert.

–
Robert Randall | xxxxx@gmail.com

Robert_Randall · March 18, 2011, 3:46pm

I forgot to include another interesting detail. If I run the SAME
code on the checked version of the OS the stack of the “stuck” stack
is different

Child-SP RetAddr : Args to Child
: Call Site
fffff8800355c4e0 fffff8800171b81f : fffffa800930a1b0 fffffa800930a1b0 0000000000000000 0000000000000000 :
storport!RaidInitializePerfOptsPassive+0x428
fffff8800355c560 fffff880017781c5 : fffffa800930a1b0 fffffa80085c8540 fffff8a001de2000 fffffa800930a1b0 :
storport!RaidAdapterStartMiniport+0x23f
fffff8800355c5d0 fffff88001777cf9 : fffffa800930a1b0 fffffa800914c6f0 fffffa800930a1b0 0000000000000000 :
storport!RaidAdapterStartDeviceIrp+0x185
fffff8800355c630 fffff88001779b5d : fffffa800930a1b0 fffffa800914c6f0 fffffa800914c6f0 fffffa800930a060 :
storport!RaidAdapterPnpIrp+0x2fd
fffff8800355c6b0 fffff8000234c19d : fffffa800930a060 fffffa800914c6f0 fffffa800914c6f0 fffffa800930a060 :
storport!RaDriverPnpIrp+0x139
fffff8800355c6f0 fffff80001abb306 : fffffa800745d9c0 fffff8800355c780 fffff80001abb078 fffffa800926a930 :
nt!PnpAsynchronousCall+0x159
fffff8800355c730 fffff800023564ea : fffffa800745d9c0 fffff80001abb078 fffffa800926a930 fffffa800745a6d0 :
nt!PnpStartDevice+0x1be
fffff8800355c7f0 fffff80002355791 : fffffa800745a6d0 fffff80000000000 fffffa8000000000 0000000000000001 :
nt!PnpStartDeviceNode+0x312
fffff8800355c8a0 fffff80002346cc8 : fffffa800745a6d0 fffffa8000000000 fffff88000000000 fffffa800745a6d0 :
nt!PipProcessStartPhase1+0xe5
fffff8800355c8f0 fffff80002345bd4 : fffffa800745a6d0 fffffa8009247700 fffff8800355cbc0 fffff80000000000 :
nt!PipProcessDevNodeTree+0x240
fffff8800355cb80 fffff80001ab94e6 : fffffa8009247700 0000000000000000 fffff800020ddc18 fffffa8006afa788 :
nt!PiRestartDevice+0xfc
fffff8800355cbe0 fffff80001c45bf6 : 0000000000000000 fffff80001ab90a4 fffff800020ddc18 fffffa8006afa680 :
nt!PnpDeviceActionWorker+0x442
fffff8800355cc80 fffff80002574c85 : 0000000000000001 fffffa8006afa680 0000000000000080 0000000000000000 :
nt!ExpWorkerThread+0x156
fffff8800355cd10 fffff80001c68f46 : fffff80001c45aa0 0000000000000001 fffff880030d3180 0000000000000000 :
nt!PspSystemThreadStartup+0x1a9
fffff8800355cd80 0000000000000000 : fffff8800355d000 fffff88003557000 fffff8800355bdb0 0000000000000000 :
nt!KiStartSystemThread+0x16

Hence my cry for the TMF file for storport.sys…

On Fri, Mar 18, 2011 at 2:42 PM, Robert Randall
wrote:
> It was suggested that I pose my problem to the list and so here it is…
>
> I am working on a STORport miniport driver for an adapter that is
> MSI-X capable.
>
> To start with I must confess that I have seen some strange results
> from the StorPortInitializePerOpts API call (PERF_CONFIGURATION_DATA
> query). ?I was getting DIFFERENT results back during my testing.
> Sometimes the NUMA information would be populate, sometimes not,
> sometimes ALL the Flags would be set, sometimes just a few. ?Last week
> (Thursday) the results of the call became consistent and now I ALWAYS
> get the same answer (see below, just 2 flags set). ?I did NOT change
> the drivers on the host during this testing time (using Intel’s
> drivers on Asus Z8PE-D12 MB). ?From the PCIe bus trace things are just
> fine.
>
> Now, I’m quite confused by the fact that StorPortInitializePerfOpts
> was returning inconsistent answers for a while and by the fact that
> it is NOT populating EVERYTHING in PERF_CONFIGURATION_DATA since the
> MB I’m running on is a NUMA architecture and has PCIe GEN 2 support.
>
> This experience has led me to not trust the hardware (opinions?). ?I
> have a new MB on order (SuperMicro), however, here is the latest
> information regarding my problem including debug output context
> information and the stack trace of the “stuck” STORport call.
>
>
>
> f FFFFFA8003CEA680 Initialize 304 Checking for MSI support using
> StorPortGetMSIInfo
> f FFFFFA8003CEA680 Initialize 319 found 256 MSI-X interrupts available
> Break instruction exception - code 80000003 (first chance)
> Initialize+0x2fd:
> fffff880053189ed cc ? ? ? ? ? ? ?int ? ? 3 ?// caused by KdBreakPoint() > 15: kd> g > f FFFFFA8003CEA680 Initialize 368 PerfOpts flags available > ? ? ? ? STOR_PERF_DPC_REDIRECTION yes > ? ? ? ? STOR_PERF_CONCURRENT_CHANNELS yes > ? ? ? ? STOR_PERF_INTERRUPT_MESSAGE_RANGES no > ? ? ? ? STOR_PERF_OPTIMIZE_FOR_COMPLETION_DURING_STARTIO no > ? ? ? ? STOR_PERF_ADV_CONFIG_LOCALITY no > f FFFFFA8003CEA680 Initialize 393 requesting use of 3 MSI-X interrupts > f FFFFFA8003CEA680 Initialize 434 StorPortInitializePerfOpts Flags > driver is USING > ? ? ? ? STOR_PERF_DPC_REDIRECTION yes > ? ? ? ? STOR_PERF_CONCURRENT_CHANNELS no > ? ? ? ? STOR_PERF_INTERRUPT_MESSAGE_RANGES no > ? ? ? ? STOR_PERF_OPTIMIZE_FOR_COMPLETION_DURING_STARTIO no > ? ? ? ? STOR_PERF_ADV_CONFIG_LOCALITY no > f FFFFFA8003CEA680 Initialize 435 Driver is using MSI-X MessageNumbers > 0 through 2 > f FFFFFA8003CEA680 Initialize 436 StorPortInitializePerfOpts was successful > f FFFFFA8003CEA680 Initialize 444 exit > > PassiveInitialize runs - omitted. > PassiveInitialize DOES configure the PCIe device resulting in the > device generating several MSI-X interrupts which are serviced by the > correct ISR. ?Note: ?I'm NOT using any DPCs yet (on purpose). ?The > PassiveInitialize thread and the first MsiInterrupt thread DO overlap. > ?I'm running with InterruptSynchronizeAll just to get things working > so there is no call to StorPort(Acquire|Release)MSISpinLock in the > ISR. ?Interrupts get serviced. > > 0 FFFFF80002056CC0 MsiInterrupt 1157 entry MessageID 0 > 0 FFFFF80002056CC0 MsiInterrupt 1207 exit > 0 FFFFF80002056CC0 MsiInterrupt 1157 entry MessageID 0 > 0 FFFFF80002056CC0 MsiInterrupt 1207 exit > > After this my miniport driver stack is stuck. ?The system is not hung > or stuck, just my driver stack. ?The system is idle expect for one > CPU. ?The stack trace of the non idle CPU is: > > Child-SP ? ? ? ? ?RetAddr ? ? ? ? ? : Args to Child > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? : Call Site > fffff8800274e5b0 fffff88005339e65 : fffffa800803c600
> 0000000000000000 fffffa8007e10100 fffffa80000f0000 : > storport!StorPortPause+0x1aa0 > fffff8800274e640 fffff8800537d6a8 : fffffa800803c600
> fffffa8007e19010 fffffa8003e45c60 fffff88005326a01 : > storport!StorPortGetUncachedExtension+0x11d5 > fffff8800274e6a0 fffff8800537d9f0 : fffffa8007e19010
> 0000000000000001 fffffa800803c600 fffffa8007e19010 : > storport!StorPortInitialize+0x3de18 > fffff8800274e6f0 fffff8800537db94 : fffffa8007e19010
> fffff88005343110 fffffa800803c4b0 fffff8800274e820 : > storport!StorPortInitialize+0x3e160 > fffff8800274e750 fffff8000228d17e : fffffa8007e19010
> fffffa8008031750 fffffa800803c4b0 fffff88002113180 : > storport!StorPortInitialize+0x3e304 > fffff8800274e790 fffff80001fc3b2d : fffffa8003fc38c0
> fffffa8008031750 fffff80001fcd250 0000000000000000 : > nt!EtwUnregister+0x33e > fffff8800274e7d0 fffff8000229c4c6 : fffff800020d2200
> fffffa8003fc4010 fffffa8008031750 fffffa8003fc41b8 : > nt!RtlAssert+0x33d > fffff8800274e890 fffff8000229c764 : fffffa8003fc4010
> fffffa8003fc003e fffffa8003fc4010 0000000000000001 : > nt!IoGetDeviceInterfaces+0x2ac6 > fffff8800274e920 fffff800022bfe96 : fffffa8003fc4010
> fffffa8003fc4010 0000000000000000 0000000000000000 : > nt!IoGetDeviceInterfaces+0x2d64 > fffff8800274e950 fffff800022c0287 : fffffa8003fc4010
> 0000000000000000 0000000000000001 fffff80002141c38 : > nt!ExCreateCallback+0x3ef6 > fffff8800274ebc0 fffff80001fcfb83 : 0000000100000003
> 0000000000000000 0000000000000001 0000000000000000 : > nt!ExCreateCallback+0x42e7 > fffff8800274ec10 fffff80001ee0a21 : fffff80001fcf870
> fffff80001e1ea01 fffff80002073600 fffffa8003cea680 : > nt!DbgSetDebugFilterState+0x2343 > fffff8800274ecb0 fffff80002173cce : 0000000000000000
> fffffa8003cea680 0000000000000080 fffffa8003cd51f0 : > nt!KeAcquireInStackQueuedSpinLockAtDpcLevel+0x2f1 > fffff8800274ed40 fffff80001ec7fe6 : fffff880022d3180
> fffffa8003cea680 fffff880022de2c0 0000000000000000 : > nt!PsCreateSystemThread+0x1da > fffff8800274ed80 0000000000000000 : fffff8800274f000
> fffff88002749000 fffff8800274d8d0 00000000`00000000 :
> nt!KeInitializeSemaphore+0x24a
>
> Any thoughts from you smart folk out there in driver land?
>
> Best,
> Robert.
>
> –
> Robert Randall | xxxxx@gmail.com
>

–
Robert Randall | xxxxx@gmail.com

Igor_Sharovar · March 18, 2011, 6:29pm

You could enable printing storport debug message. By default, the check build version of storport doesn’t print any message. For such problem I would also install the check build of PCI.sys and enable debug messages for this driver also.

Igor Sharovar

Alex_Grig · March 19, 2011, 10:22am

You need to load proper debug symbols for the system files. The printed offsets are all bogus.

Robert_Randall · March 19, 2011, 5:01pm

BOTH stack traces are bogus?

On Mar 19, 2011, at 9:20 AM, xxxxx@broadcom.com wrote:

You need to load proper debug symbols for the system files. The printed offsets are all bogus.

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Alex_Grig · March 19, 2011, 7:41pm

When you see large offsets from function names, that means the debugger is using a nearest exported symbol. Do

!lmi storport.sys

to see if the symbols are loaded for it.

Robert_Randall · March 20, 2011, 7:57am

Thanks. HOW do I enable debug messages in the checked build of storport.sys
and pci.sys?

I have looked for an appropriate module global symbol name (dt storport!*)
but didn’t find anything.

However, I have also been told my symbols are not matching what is
executing.

TIA,
Robert Randall

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@hotmail.com
Sent: Friday, March 18, 2011 5:28 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] STORport hang while attempting to use MSI-X

You could enable printing storport debug message. By default, the check
build version of storport doesn’t print any message. For such problem I
would also install the check build of PCI.sys and enable debug messages for
this driver also.

Igor Sharovar

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Igor_Sharovar · March 20, 2011, 9:36pm

>Thanks. HOW do I enable debug messages in the checked build of >storport.sys and pci.sys?
To enable storport.sys debug message you need to set up a variable nt!Kd_STORMINIPORT_Mask to 0xFFFFFFFF. If you correctly set up the variable you should see debug messages in WinDbg which start with word “STOR”.
To enable debug messages for pci.sys you need to set up a variable nt!Kd_WIN2000_mask( or nt!Kd_WIN_2000_mask, I don’t remember exactly) also to 0xFFFFFFFF. But be aware that this variable could enable debug messages for others check build drivers of Windows. I would recommend to use only necessary drivers to debug. Otherwise you would get a lot of messages in WinDbg.
When you will analyze pci.sys debug trace pay attention to initialization PCI resources of your driver. This trace helped me to solve a problem similar to yours.

Igor Sharovar

Robert_Randall · March 22, 2011, 1:35pm

Many thanks. I was able to turn on the debug output for the pci bus
driver and for storport.

I belive I have a correct stack trace now. I am still experiencing
the same problem.

I am going to instrument my PnP routines a bit more. The request for
a minor of 18 is curious.

I can’t find an IRP_MN_PNP of 0x18.

The stack appears to be stuck waiting for a DPC as part of the
PerfOpts initialization which
appears to be occuring at passive given the function name.

I have heard from other sources that the Flag
STOR_PERF_INTERRUPT_MESSAGE_RANGES
is MANDATORY for MSI-X messages to work properly.

STORport is NOT setting this flag when the PerfOpts are queried.

Can anyone confirm what I have heard?

Thanks for your attention.

Best regards,
Robert Randall

Debug output:

fffff800`02080490 cc int 3
kd> .reload /f
Connected to Windows 7 7601 x64 target at (Tue Mar 22 11:25:28.456
2011 (GMT-5)), ptr64 TRUE
Loading Kernel Symbols
…*** ERROR: Module load completed
but symbols could not be loaded for spldr.sys
…
Loading User Symbols

kd> x nt!Kd_STOR*
fffff80002211214 nt!Kd_STORMINIPORT_Mask = <no type information> fffff800022110d8 nt!Kd_STORPROP_Mask =
fffff80002211210 nt!Kd_STORPORT_Mask = <no type information> fffff800022112d0 nt!Kd_STORVSP_Mask =
kd> ed nt!Kd_STORPORT_Mask 0xffffffff
kd> ed nt!Kd_STORMINIPORT_Mask 0xffffffff
kd> g

After this there is a bit of noise since the iSCSI initiator is installed…

2 FFFFFA8003CE4040 DriverEntry 62 Storport Driver - Built Mar 21 2011 16:27:22
2 FFFFFA8003CE4040 DriverEntry 65 System has 0x1 NUMA nodes
2 FFFFFA8003CE4040 DriverEntry 67 Driver running on NUMA node 0x0 NUMA
2 FFFFFA8003CE4040 DriverEntry 70 Driver running on processor 0x2 in group 0x0
STOR: RaidPortInitialize: DriverObject=FFFFFA80044795F0,
RegistryPath=FFFFFA800447C000, HwInitData=FFFFF880027551D0,
HwContext=0000000000000000
STOR: AddDevice: DriverObject FFFFFA80044795F0, PhysicalDeviceObject
FFFFFA800479C9C0
STOR: DeviceObject FFFFFA800447C060, Irp FFFFFA8007B319A0 PnP, Minor 18
STOR: Adapter FFFFFA800447C1B0, Irp FFFFFA8007B319A0, Pnp 18
STOR: Adapter FFFFFA800447C1B0, Irp FFFFFA8007B319A0, Pnp 18 ret = c00000bb
STOR: DeviceObject FFFFFA800447C060, Irp FFFFFA8007B319A0 PnP, ret = c00000bb
STOR: DeviceObject FFFFFA800447C060, Irp FFFFFA8007B319A0 PnP, Minor d
STOR: Adapter FFFFFA800447C1B0, Irp FFFFFA8007B319A0, Pnp d
STOR: Adapter FFFFFA800447C1B0, Irp FFFFFA8007B319A0, Pnp d ret = 0
STOR: DeviceObject FFFFFA800447C060, Irp FFFFFA8007B319A0 PnP, ret = 00000000
STOR: DeviceObject FFFFFA800447C060, Irp FFFFFA8007B319A0 PnP, Minor 0
STOR: Adapter FFFFFA800447C1B0, Irp FFFFFA8007B319A0, Pnp 0
STOR: Adapter FFFFFA800447C1B0, Irp FFFFFA8007B319A0, Pnp StartDevice
0 FFFFFA8003CE4040 FindAdapter 157 entry
0 FFFFFA8003CE4040 FindAdapter 281 exit
3 FFFFFA8003CE4040 Initialize 309 entry
3 FFFFFA8003CE4040 Initialize 310 assuming no MSI support
3 FFFFFA8003CE4040 Initialize 312 Checking for MSI support using
StorPortGetMSIInfo
STOR: StorPortExtendedFunction: Function code: 13
STOR: StorPortExtendedFunction: Function code: 13
STOR: StorPortExtendedFunction: Function code: 13

3 FFFFFA8003CE4040 Initialize 327 found 256 MSI-X interrupts available
STOR: StorPortExtendedFunction: Function code: 14
Break instruction exception - code 80000003 (first chance)
Initialize+0x2bb:
fffff880053eaa1b cc int 3 3: kd> p Initialize+0x2bc: fffff880053eaa1c 488b442440 mov rax,qword ptr [rsp+40h]
3: kd> p
3 FFFFFA8003CE4040 Initialize 380 PerfOpts flags available
STOR_PERF_DPC_REDIRECTION yes
STOR_PERF_CONCURRENT_CHANNELS yes
STOR_PERF_INTERRUPT_MESSAGE_RANGES no
STOR_PERF_OPTIMIZE_FOR_COMPLETION_DURING_STARTIO no
STOR_PERF_ADV_CONFIG_LOCALITY no
Initialize+0x3fb:
fffff880053eab5b 488b442440 mov rax,qword ptr [rsp+40h] 3: kd> p Initialize+0x488: fffff880053eabe8 ba28000000 mov edx,28h
3: kd> g
3 FFFFFA8003CE4040 Initialize 413 requesting use of 3 MSI-X interrupts
STOR: StorPortExtendedFunction: Function code: 14
3 FFFFFA8003CE4040 Initialize 454 StorPortInitializePerfOpts Flags
driver is USING
STOR_PERF_DPC_REDIRECTION yes
STOR_PERF_CONCURRENT_CHANNELS no
STOR_PERF_INTERRUPT_MESSAGE_RANGES no
STOR_PERF_OPTIMIZE_FOR_COMPLETION_DURING_STARTIO no
STOR_PERF_ADV_CONFIG_LOCALITY no
3 FFFFFA8003CE4040 Initialize 455 Driver is using MSI-X MessageNumbers
0 through 2
3 FFFFFA8003CE4040 Initialize 456 StorPortInitializePerfOpts was successful
3 FFFFFA8003CE4040 Initialize 468 setting up passive init
3 FFFFFA8003CE4040 Initialize 475 exit
3 FFFFFA8003CE4040 PassiveInitialize 502 entry
6 FFFFF88001FEE2C0 MsiInterrupt 1192 entry MessageID 0
6 FFFFF88001FEE2C0 MsiInterrupt 1246 exit
6 FFFFF88001FEE2C0 MsiInterrupt 1192 entry MessageID 0
6 FFFFF88001FEE2C0 MsiInterrupt 1246 exit
6 FFFFF88001FEE2C0 MsiInterrupt 1192 entry MessageID 0
6 FFFFF88001FEE2C0 MsiInterrupt 1246 exit
6 FFFFF88001FEE2C0 MsiInterrupt 1192 entry MessageID 0
6 FFFFF88001FEE2C0 MsiInterrupt 1246 exit
6 FFFFF88001FEE2C0 MsiInterrupt 1192 entry MessageID 0
3 FFFFFA8003CE4040 PassiveInitialize 529 exit
6 FFFFF88001FEE2C0 MsiInterrupt 1246 exit
6 FFFFF88001FEE2C0 MsiInterrupt 1192 entry MessageID 0
6 FFFFF88001FEE2C0 MsiInterrupt 1246 exit
nt!RtlpBreakWithStatusInstruction:
fffff80002085490 cc int 3 0: kd> .reload /f Connected to Windows 7 7601 x64 target at (Tue Mar 22 11:45:30.580 2011 (GMT-5)), ptr64 TRUE Loading Kernel Symbols ..................................... ***ERROR: Module load completed but symbols could not be loaded for spldr.sys .......................... ............*** ERROR: Module load completed but symbols could not be loaded for astgrp.sys .................................. ***WARNING: Unable to verify timestamp for ASTGRV.dll *** ERROR: Module load completed but symbols could not be loaded for ASTGRV.dll .......... ***ERROR: Module load completed but symbols could not be loaded for peauth.sys .*** ERROR: Module load completed but symbols could not be loaded for secdrv.SYS ..... Loading User Symbols Loading unloaded module list ...... 14: kd> ~3 3: kd> kv Child-SP RetAddr : Args to Child : Call Site fffff88002755558 fffff8800344ad26 : 0000000000000001
0000000000002000 fffffa800447c1b0 0000000000000000 : nt!KiDpcInterrupt+0x10 (TrapFrame @ fffff880027553f0)
fffff88002755580 fffff8800344a81f : fffffa800447c1b0 fffffa800447c1b0 0000000000000000 0000000000000000 :
storport!RaidInitializePerfOptsPassive+0x3ca
fffff88002755600 fffff880034a71c5 : fffffa800447c1b0 fffffa8003dcd940 fffff8a001bd2000 fffffa800447c1b0 :
storport!RaidAdapterStartMiniport+0x23f
fffff88002755670 fffff880034a6cf9 : fffffa800447c1b0 fffffa8007b319a0 fffffa800447c1b0 0000000000000000 :
storport!RaidAdapterStartDeviceIrp+0x185
fffff880027556d0 fffff880034a8b5d : fffffa800447c1b0 fffffa8007b319a0 fffffa8007b319a0 fffffa800447c060 :
storport!RaidAdapterPnpIrp+0x2fd
fffff88002755750 fffff8000244417e : fffffa800447c060 fffffa8007b319a0 fffffa8007b319a0 fffffa800447c060 :
storport!RaDriverPnpIrp+0x139
fffff88002755790 fffff8000217ab2d : fffffa800479c9c0 fffffa8007c580e0 fffff80002184250 0000000000000000 :
nt!PnpAsynchronousCall+0xce
fffff880027557d0 fffff800024534c6 : fffff80002289200 fffffa800479b6e0 fffffa8007c580e0 fffffa800479b888 :
nt!PnpStartDevice+0x11d
fffff88002755890 fffff80002453764 : fffffa800479b6e0 fffffa800479003e fffffa800479b6e0 0000000000000001 :
nt!PnpStartDeviceNode+0x156
fffff88002755920 fffff80002476e96 : fffffa800479b6e0 fffffa800479b6e0 0000000000000000 0000000000000000 :
nt!PipProcessStartPhase1+0x74
fffff88002755950 fffff80002477287 : fffffa800479b6e0 0000000000000000 0000000000000001 fffff800022f8c38 :
nt!PipProcessDevNodeTree+0x296
fffff88002755bc0 fffff80002186b83 : 0000000100000003 0000000000000000 0000000000000001 0000000000000000 :
nt!PiRestartDevice+0xc7
fffff88002755c10 fffff80002097a21 : fffff80002186870 fffff80002383f01 fffffa8003ce4000 fffffa8003ce4040 :
nt!PnpDeviceActionWorker+0x313
fffff88002755cb0 fffff8000232acce : 0000000000000000 fffffa8003ce4040 0000000000000080 fffffa8003cd1040 :
nt!ExpWorkerThread+0x111
fffff88002755d40 fffff8000207efe6 : fffff880020a3180 fffffa8003ce4040 fffff880020ae2c0 0000000000000000 :
nt!PspSystemThreadStartup+0x5a
fffff88002755d80 0000000000000000 : fffff88002756000 fffff88002750000 fffff88002753210 0000000000000000 :
nt!KxStartSystemThread+0x167

–
Robert Randall | xxxxx@gmail.com

Robert_Randall · March 22, 2011, 1:50pm

Follow up on the PnP questions. The port driver must be handling them
because they never hit my StartIo function.

On Tue, Mar 22, 2011 at 12:34 PM, Robert Randall
wrote:
> Many thanks. ?I was able to turn on the debug output for the pci bus
> driver and for storport.
>
> I belive I have a correct stack trace now. ?I am still experiencing
> the same problem.
>
> I am going to instrument my PnP routines a bit more. ?The request for
> a minor of 18 is curious.
>
> I can’t find an IRP_MN_PNP of 0x18.
>
> The stack appears to be stuck waiting for a DPC as part of the
> PerfOpts initialization which
> appears to be occuring at passive given the function name.
>
> I have heard from other sources that the Flag
> STOR_PERF_INTERRUPT_MESSAGE_RANGES
> is MANDATORY for MSI-X messages to work properly.
>
> STORport is NOT setting this flag when the PerfOpts are queried.
>
> Can anyone confirm what I have heard?
>
> Thanks for your attention.
>
> Best regards,
> Robert Randall
>
> Debug output:
>
> fffff80002080490 cc ? ? ? ? ? ? ?int ? ? 3 > kd> .reload /f > Connected to Windows 7 7601 x64 target at (Tue Mar 22 11:25:28.456 > 2011 (GMT-5)), ptr64 TRUE > Loading Kernel Symbols > ......................................***ERROR: Module load completed > but symbols could not be loaded for spldr.sys > .... > Loading User Symbols > > kd> x nt!Kd_STOR* > fffff80002211214 nt!Kd_STORMINIPORT_Mask =
> fffff800022110d8 nt!Kd_STORPROP_Mask = <no type information> > fffff80002211210 nt!Kd_STORPORT_Mask =
> fffff800022112d0 nt!Kd_STORVSP_Mask = <no type information> > kd> ed nt!Kd_STORPORT_Mask 0xffffffff > kd> ed nt!Kd_STORMINIPORT_Mask 0xffffffff > kd> g > > After this there is a bit of noise since the iSCSI initiator is installed... > > 2 FFFFFA8003CE4040 DriverEntry 62 Storport Driver - Built Mar 21 2011 16:27:22 > 2 FFFFFA8003CE4040 DriverEntry 65 System has 0x1 NUMA nodes > 2 FFFFFA8003CE4040 DriverEntry 67 Driver running on NUMA node 0x0 NUMA > 2 FFFFFA8003CE4040 DriverEntry 70 Driver running on processor 0x2 in group 0x0 > STOR: RaidPortInitialize: DriverObject=FFFFFA80044795F0, > RegistryPath=FFFFFA800447C000, HwInitData=FFFFF880027551D0, > HwContext=0000000000000000 > STOR: AddDevice: DriverObject FFFFFA80044795F0, PhysicalDeviceObject > FFFFFA800479C9C0 > STOR: DeviceObject FFFFFA800447C060, Irp FFFFFA8007B319A0 PnP, Minor 18 > STOR: Adapter FFFFFA800447C1B0, Irp FFFFFA8007B319A0, Pnp 18 > STOR: Adapter FFFFFA800447C1B0, Irp FFFFFA8007B319A0, Pnp 18 ret = c00000bb > STOR: DeviceObject FFFFFA800447C060, Irp FFFFFA8007B319A0 PnP, ret = c00000bb > STOR: DeviceObject FFFFFA800447C060, Irp FFFFFA8007B319A0 PnP, Minor d > STOR: Adapter FFFFFA800447C1B0, Irp FFFFFA8007B319A0, Pnp d > STOR: Adapter FFFFFA800447C1B0, Irp FFFFFA8007B319A0, Pnp d ret = 0 > STOR: DeviceObject FFFFFA800447C060, Irp FFFFFA8007B319A0 PnP, ret = 00000000 > STOR: DeviceObject FFFFFA800447C060, Irp FFFFFA8007B319A0 PnP, Minor 0 > STOR: Adapter FFFFFA800447C1B0, Irp FFFFFA8007B319A0, Pnp 0 > STOR: Adapter FFFFFA800447C1B0, Irp FFFFFA8007B319A0, Pnp StartDevice > 0 FFFFFA8003CE4040 FindAdapter 157 entry > 0 FFFFFA8003CE4040 FindAdapter 281 exit > 3 FFFFFA8003CE4040 Initialize 309 entry > 3 FFFFFA8003CE4040 Initialize 310 assuming no MSI support > 3 FFFFFA8003CE4040 Initialize 312 Checking for MSI support using > StorPortGetMSIInfo > STOR: StorPortExtendedFunction: Function code: 13 > STOR: StorPortExtendedFunction: Function code: 13 > STOR: StorPortExtendedFunction: Function code: 13 > <repeats times for each msi-x message found which is correct> > 3 FFFFFA8003CE4040 Initialize 327 found 256 MSI-X interrupts available > STOR: StorPortExtendedFunction: Function code: 14 > Break instruction exception - code 80000003 (first chance) > Initialize+0x2bb: > fffff880053eaa1b cc ? ? ? ? ? ? ?int ? ? 3
> 3: kd> p
> Initialize+0x2bc:
> fffff880053eaa1c 488b442440 ? ? ?mov ? ? rax,qword ptr [rsp+40h] > 3: kd> p > 3 FFFFFA8003CE4040 Initialize 380 PerfOpts flags available > ? ? ? ? STOR_PERF_DPC_REDIRECTION yes > ? ? ? ? STOR_PERF_CONCURRENT_CHANNELS yes > ? ? ? ? STOR_PERF_INTERRUPT_MESSAGE_RANGES no > ? ? ? ? STOR_PERF_OPTIMIZE_FOR_COMPLETION_DURING_STARTIO no > ? ? ? ? STOR_PERF_ADV_CONFIG_LOCALITY no > Initialize+0x3fb: > fffff880053eab5b 488b442440 ? ? ?mov ? ? rax,qword ptr [rsp+40h]
> 3: kd> p
> Initialize+0x488:
> fffff880053eabe8 ba28000000 ? ? ?mov ? ? edx,28h > 3: kd> g > 3 FFFFFA8003CE4040 Initialize 413 requesting use of 3 MSI-X interrupts > STOR: StorPortExtendedFunction: Function code: 14 > 3 FFFFFA8003CE4040 Initialize 454 StorPortInitializePerfOpts Flags > driver is USING > ? ? ? ? STOR_PERF_DPC_REDIRECTION yes > ? ? ? ? STOR_PERF_CONCURRENT_CHANNELS no > ? ? ? ? STOR_PERF_INTERRUPT_MESSAGE_RANGES no > ? ? ? ? STOR_PERF_OPTIMIZE_FOR_COMPLETION_DURING_STARTIO no > ? ? ? ? STOR_PERF_ADV_CONFIG_LOCALITY no > 3 FFFFFA8003CE4040 Initialize 455 Driver is using MSI-X MessageNumbers > 0 through 2 > 3 FFFFFA8003CE4040 Initialize 456 StorPortInitializePerfOpts was successful > 3 FFFFFA8003CE4040 Initialize 468 setting up passive init > 3 FFFFFA8003CE4040 Initialize 475 exit > 3 FFFFFA8003CE4040 PassiveInitialize 502 entry > 6 FFFFF88001FEE2C0 MsiInterrupt 1192 entry MessageID 0 > 6 FFFFF88001FEE2C0 MsiInterrupt 1246 exit > 6 FFFFF88001FEE2C0 MsiInterrupt 1192 entry MessageID 0 > 6 FFFFF88001FEE2C0 MsiInterrupt 1246 exit > 6 FFFFF88001FEE2C0 MsiInterrupt 1192 entry MessageID 0 > 6 FFFFF88001FEE2C0 MsiInterrupt 1246 exit > 6 FFFFF88001FEE2C0 MsiInterrupt 1192 entry MessageID 0 > 6 FFFFF88001FEE2C0 MsiInterrupt 1246 exit > 6 FFFFF88001FEE2C0 MsiInterrupt 1192 entry MessageID 0 > 3 FFFFFA8003CE4040 PassiveInitialize 529 exit > 6 FFFFF88001FEE2C0 MsiInterrupt 1246 exit > 6 FFFFF88001FEE2C0 MsiInterrupt 1192 entry MessageID 0 > 6 FFFFF88001FEE2C0 MsiInterrupt 1246 exit > nt!RtlpBreakWithStatusInstruction: > fffff80002085490 cc ? ? ? ? ? ? ?int ? ? 3
> 0: kd> .reload /f
> Connected to Windows 7 7601 x64 target at (Tue Mar 22 11:45:30.580
> 2011 (GMT-5)), ptr64 TRUE
> Loading Kernel Symbols
> …*** ERROR: Module load completed
> but symbols could not be loaded for spldr.sys
> …
> … ERROR: Module load completed but symbols could not be
> loaded for astgrp.sys
> … WARNING: Unable to verify
> timestamp for ASTGRV.dll
> ERROR: Module load completed but symbols could not be loaded for ASTGRV.dll
> … ERROR: Module load completed but symbols could not be
> loaded for peauth.sys
> .*** ERROR: Module load completed but symbols could not be loaded for secdrv.SYS
> …
> Loading User Symbols
>
> Loading unloaded module list
> …
> 14: kd> ~3
> 3: kd> kv
> Child-SP ? ? ? ? ?RetAddr ? ? ? ? ? : Args to Child
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? : Call Site
> fffff88002755558 fffff8800344ad26 : 0000000000000001 > 0000000000002000 fffffa800447c1b0 0000000000000000 :
> nt!KiDpcInterrupt+0x10 (TrapFrame @ fffff880027553f0) > fffff88002755580 fffff8800344a81f : fffffa800447c1b0
> fffffa800447c1b0 0000000000000000 0000000000000000 : > storport!RaidInitializePerfOptsPassive+0x3ca > fffff88002755600 fffff880034a71c5 : fffffa800447c1b0
> fffffa8003dcd940 fffff8a001bd2000 fffffa800447c1b0 : > storport!RaidAdapterStartMiniport+0x23f > fffff88002755670 fffff880034a6cf9 : fffffa800447c1b0
> fffffa8007b319a0 fffffa800447c1b0 0000000000000000 : > storport!RaidAdapterStartDeviceIrp+0x185 > fffff880027556d0 fffff880034a8b5d : fffffa800447c1b0
> fffffa8007b319a0 fffffa8007b319a0 fffffa800447c060 : > storport!RaidAdapterPnpIrp+0x2fd > fffff88002755750 fffff8000244417e : fffffa800447c060
> fffffa8007b319a0 fffffa8007b319a0 fffffa800447c060 : > storport!RaDriverPnpIrp+0x139 > fffff88002755790 fffff8000217ab2d : fffffa800479c9c0
> fffffa8007c580e0 fffff80002184250 0000000000000000 : > nt!PnpAsynchronousCall+0xce > fffff880027557d0 fffff800024534c6 : fffff80002289200
> fffffa800479b6e0 fffffa8007c580e0 fffffa800479b888 : > nt!PnpStartDevice+0x11d > fffff88002755890 fffff80002453764 : fffffa800479b6e0
> fffffa800479003e fffffa800479b6e0 0000000000000001 : > nt!PnpStartDeviceNode+0x156 > fffff88002755920 fffff80002476e96 : fffffa800479b6e0
> fffffa800479b6e0 0000000000000000 0000000000000000 : > nt!PipProcessStartPhase1+0x74 > fffff88002755950 fffff80002477287 : fffffa800479b6e0
> 0000000000000000 0000000000000001 fffff800022f8c38 : > nt!PipProcessDevNodeTree+0x296 > fffff88002755bc0 fffff80002186b83 : 0000000100000003
> 0000000000000000 0000000000000001 0000000000000000 : > nt!PiRestartDevice+0xc7 > fffff88002755c10 fffff80002097a21 : fffff80002186870
> fffff80002383f01 fffffa8003ce4000 fffffa8003ce4040 : > nt!PnpDeviceActionWorker+0x313 > fffff88002755cb0 fffff8000232acce : 0000000000000000
> fffffa8003ce4040 0000000000000080 fffffa8003cd1040 : > nt!ExpWorkerThread+0x111 > fffff88002755d40 fffff8000207efe6 : fffff880020a3180
> fffffa8003ce4040 fffff880020ae2c0 0000000000000000 : > nt!PspSystemThreadStartup+0x5a > fffff88002755d80 0000000000000000 : fffff88002756000
> fffff88002750000 fffff88002753210 00000000`00000000 :
> nt!KxStartSystemThread+0x167
>
>
> –
> Robert Randall | xxxxx@gmail.com
>

–
Robert Randall | xxxxx@gmail.com

Alex_Grig · March 22, 2011, 4:34pm

At what point you enable device interrupts in your driver?

Igor_Sharovar · March 22, 2011, 8:37pm

>3 FFFFFA8003CE4040 Initialize 310 assuming no MSI support

Did you enable MSI support in PORT_CONFIGURATION_INFORMATION? You must do it in Hw_FindAdapter. You also need enable MSI in you *.inf file.

Igor Sharovar

Robert_Randall · March 23, 2011, 9:28am

During Initialize and activity during PassiveInitialize causes
interrupts and they do cause my MsiInterrupt ISR to be called.

On Tue, Mar 22, 2011 at 3:33 PM, wrote:
> At what point you enable device interrupts in your driver?
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>

–
Robert Randall | xxxxx@gmail.com

Robert_Randall · March 23, 2011, 9:32am

Yes. I have what I believe to be the correct entries in the INF file
and I have enabled MSI in the port configuration during FindAdapter:

ConfigInfo->InterruptSynchronizationMode = InterruptSynchronizeAll;
ConfigInfo->HwMSInterruptRoutine = MsiInterrupt;

From the INF file:

HKR, “ScsiPort”, “NeedsSystemShutdownNotification”, 0x00010001, 1
HKR, Interrupt Management, 0x00000010
HKR, Interrupt Management\MessageSignaledInterruptProperties, 0x00000010
HKR, Interrupt Management\MessageSignaledInterruptProperties,
MSISupported, 0x00010001, 1
HKR, Interrupt Management\MessageSignaledInterruptProperties,
MessageNumberLimit, 0x00010001, 256
HKR, Interrupt Management\Affinity Policy, 0x00000010
HKR, Interrupt Management\Affinity Policy, DevicePolicy, 0x00010001, 0x00000001
HKR, Interrupt Management\Affinity Policy, DevicePriority, 0x00010001,
0x00000002

Thanks again!

Robert Randall

On Tue, Mar 22, 2011 at 7:35 PM, wrote:
>>3 FFFFFA8003CE4040 Initialize 310 assuming no MSI support
>
> Did you enable MSI support in PORT_CONFIGURATION_INFORMATION? You must do it in Hw_FindAdapter. You also need enable MSI in you *.inf file.
>
> Igor Sharovar
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>

–
Robert Randall | xxxxx@gmail.com

Alex_Grig · March 23, 2011, 10:50am

What’s wrong with interrupts happening during PassiveInitialize? At that time the hardware is assumed to be up. Hardware should be fully initialized (at least to be able to issue interrupts) during HwStorInitalize which happens on DIRQL.

Also, what MSI synchronization mode do you specify?

Robert_Randall · March 23, 2011, 4:39pm

I agree on the interrupts but I’m questioning everything at the moment.

I’m using InterruptSynchronizeAll until I can get the darn things to
work at all.

On Wed, Mar 23, 2011 at 9:48 AM, wrote:
> What’s wrong with interrupts happening during PassiveInitialize? At that time the hardware is assumed to be up. Hardware should be fully initialized (at least to be able to issue interrupts) during HwStorInitalize which happens on DIRQL.
>
> Also, what MSI synchronization mode do you specify?
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>

–
Robert Randall | xxxxx@gmail.com

Robert_Randall · March 24, 2011, 12:37pm

After opening a case with PSS and getting some ‘guidance’ on what to
do the mystery was solved.

The documentation is missing some very important content. When
calling StorPortInitializePerfOpts the PERF_CONFIGURATION_DATA you
pass in MUST have two fields set; Size and Version. I was missing
Version. You can find the appropriate version constant in storport.h
(depends on WDK/DDK version).

Thanks all for the assistance.

I hope this keeps someone else from going through this painful process ;=)

Best,
Robert Randall

On Wed, Mar 23, 2011 at 3:38 PM, Robert Randall
wrote:
> I agree on the interrupts but I’m questioning everything at the moment.
>
> I’m using InterruptSynchronizeAll until I can get the darn things to
> work at all.
>
> On Wed, Mar 23, 2011 at 9:48 AM, ? wrote:
>> What’s wrong with interrupts happening during PassiveInitialize? At that time the hardware is assumed to be up. Hardware should be fully initialized (at least to be able to issue interrupts) during HwStorInitalize which happens on DIRQL.
>>
>> Also, what MSI synchronization mode do you specify?
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>>
>
>
>
> –
> Robert Randall | xxxxx@gmail.com
>

–
Robert Randall | xxxxx@gmail.com