SRB data changed after complete

Vincent_Jin · August 31, 2018, 5:22am

This is a storport driver. It works fine on Server 2012.
But the data is modified after StorPortNotification on Server 2008 R2.

Following is the details:

A simple test application open the block device and read first 4K data. First 4k is set to ‘0x5a’ before read.
Before StorPortNotification, we get virtual address of this srb by StorPortGetSystemAddress. The 4k data in returned virtual address are all ‘0x5a’.
I set a break point when write to address (systemaddress+0x5b0). Got following stack. 0x5b0 is the place where data is changed.
===============================================

Child-SP RetAddr Call Site

00 fffff880`025c5c38 fffff800`023f5e15 hal!memcpy+0x266
01 fffff880`025c5c40 fffff800`023f569a hal!HalpDmaSyncMapBuffers+0x1bd
02 fffff880`025c5cf0 fffff800`023f883f hal!HalpDmaFlushScatterTransfer+0xae
03 fffff880`025c5d40 fffff800`023f77d5 hal!IoFlushAdapterBuffers+0xaf
04 fffff880`025c5d80 fffff880`03b6c787 hal!HalPutScatterGatherList+0x85
05 fffff880`025c5de0 fffff880`03b7344e storport!RaidUnitCompleteRequest+0x1b7
06 fffff880`025c5ec0 fffff800`01e94b1c storport!RaidpAdapterRedirectDpcRoutine+0x4e
07 fffff880`025c5f00 fffff800`01e8c165 nt!KiRetireDpcList+0x1bc
08 fffff880`025c5fb0 fffff800`01e8bf7c nt!KyRetireDpcList+0x5
09 fffff880`02d93150 fffff800`01ed531c nt!KiDispatchInterruptContinue
0a fffff880`02d93180 fffff880`03b6e491 nt!KiDpcInterrupt+0xcc
0b fffff880`02d93310 fffff880`03b4a72c storport! ?? ::FNODOBFM::`string'+0xb58 0c fffff880`02d93390 fffff880`03b49756 vfd!VFD_STOPROT_NOTIFICATION+0x8c 0d fffff880`02d933f0 fffff880`03b3b250 vfd!vfd_wbio_complete+0x576 0e fffff880`02d934e0 fffff880`03b3babc vfd!vfd_vbio_end+0x130 0f fffff880`02d93540 fffff880`03b3c0f2 vfd!vfd_vbio_put+0x22c 10 fffff880`02d935c0 fffff880`03b30db9 vfd!vfd_workunit_vbio_put+0x152 11 fffff880`02d93770 fffff880`03b2bb10 vfd!vfd_blk_end_io_async+0x1529 12 fffff880`02d93940 fffff880`03b3e8d7 vfd!vfd_krnl_rw_pmem_end_io_async+0x100 13 fffff880`02d939b0 fffff880`03b3e0aa vfd!vfd_cmd_complete+0x817 14 fffff880`02d93ab0 fffff880`03b28748 vfd!vfd_callback_worker+0x10a 15 fffff880`02d93b00 fffff800`0217ff33 vfd!vfdio_workitem_general_func+0x28 16 fffff880`02d93b40 fffff800`01e93a21 nt!IopProcessWorkItem+0x23 17 fffff880`02d93b70 fffff800`02126cce nt!ExpWorkerThread+0x111 18 fffff880`02d93c00 fffff800`01e7afe6 nt!PspSystemThreadStartup+0x5a 19 fffff880`02d93c40 00000000`00000000 nt!KiStartSystemThread+0x16

Data is changed to following. Correct value should be ‘0x5a’.

00000000: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
… …
000005a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
000005b0: 00 00 00 04 cc cc cc c0 00 00 02 00 5a 5a 5a 5a
000005c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
…
00000ff0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a

According to the stack, I’m very confused why driver stack do memory copy in put scatter and gather list? Also Why the data is modified?

Mark_Roddy · September 1, 2018, 5:07pm

If there is a memcpy going on then you are using a bounce buffer for dma ,
which might explain why you think you are seeing your ‘data buffer’ being
modified after completion.

Mark Roddy

On Fri, Aug 31, 2018 at 5:22 AM xxxxx@gmail.com
wrote:

> This is a storport driver. It works fine on Server 2012.
> But the data is modified after StorPortNotification on Server 2008 R2.
>
> Following is the details:
> 1. A simple test application open the block device and read first 4K
> data. First 4k is set to ‘0x5a’ before read.
> 2. Before StorPortNotification, we get virtual address of this srb by
> StorPortGetSystemAddress. The 4k data in returned virtual address are all
> ‘0x5a’.
> 3. I set a break point when write to address (systemaddress+0x5b0). Got
> following stack. 0x5b0 is the place where data is changed.
> ===============================================
> # Child-SP RetAddr Call Site
> 00 fffff880025c5c38 fffff800023f5e15 hal!memcpy+0x266
> 01 fffff880025c5c40 fffff800023f569a hal!HalpDmaSyncMapBuffers+0x1bd
> 02 fffff880025c5cf0 fffff800023f883f hal!HalpDmaFlushScatterTransfer+0xae
> 03 fffff880025c5d40 fffff800023f77d5 hal!IoFlushAdapterBuffers+0xaf
> 04 fffff880025c5d80 fffff88003b6c787 hal!HalPutScatterGatherList+0x85
> 05 fffff880025c5de0 fffff88003b7344e
> storport!RaidUnitCompleteRequest+0x1b7
> 06 fffff880025c5ec0 fffff80001e94b1c
> storport!RaidpAdapterRedirectDpcRoutine+0x4e
> 07 fffff880025c5f00 fffff80001e8c165 nt!KiRetireDpcList+0x1bc
> 08 fffff880025c5fb0 fffff80001e8bf7c nt!KyRetireDpcList+0x5
> 09 fffff88002d93150 fffff80001ed531c nt!KiDispatchInterruptContinue
> 0a fffff88002d93180 fffff88003b6e491 nt!KiDpcInterrupt+0xcc
> 0b fffff88002d93310 fffff88003b4a72c storport! ??
> ::FNODOBFM::string'+0xb58 > 0c fffff88002d93390 fffff88003b49756 vfd!VFD_STOPROT_NOTIFICATION+0x8c > 0d fffff88002d933f0 fffff88003b3b250 vfd!vfd_wbio_complete+0x576 > 0e fffff88002d934e0 fffff88003b3babc vfd!vfd_vbio_end+0x130 > 0f fffff88002d93540 fffff88003b3c0f2 vfd!vfd_vbio_put+0x22c > 10 fffff88002d935c0 fffff88003b30db9 vfd!vfd_workunit_vbio_put+0x152 > 11 fffff88002d93770 fffff88003b2bb10 vfd!vfd_blk_end_io_async+0x1529 > 12 fffff88002d93940 fffff88003b3e8d7 > vfd!vfd_krnl_rw_pmem_end_io_async+0x100 > 13 fffff88002d939b0 fffff88003b3e0aa vfd!vfd_cmd_complete+0x817 > 14 fffff88002d93ab0 fffff88003b28748 vfd!vfd_callback_worker+0x10a > 15 fffff88002d93b00 fffff8000217ff33 > vfd!vfdio_workitem_general_func+0x28 > 16 fffff88002d93b40 fffff80001e93a21 nt!IopProcessWorkItem+0x23 > 17 fffff88002d93b70 fffff80002126cce nt!ExpWorkerThread+0x111 > 18 fffff88002d93c00 fffff80001e7afe6 nt!PspSystemThreadStartup+0x5a > 19 fffff88002d93c40 00000000`00000000 nt!KiStartSystemThread+0x16
> ================================================
> Data is changed to following. Correct value should be ‘0x5a’.
> ================================================
> 00000000: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
> … …
> 000005a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 000005b0: 00 00 00 04 cc cc cc c0 00 00 02 00 5a 5a 5a 5a
> 000005c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
> …
> 00000ff0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
> ================================================
>
> According to the stack, I’m very confused why driver stack do memory copy
> in put scatter and gather list? Also Why the data is modified?
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: <
> http://www.osronline.com/showlists.cfm?list=ntdev>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer>
></http:>

Vincent_Jin · September 2, 2018, 1:13am

In HwFindAdapter function, PPORT_CONFIGURATION_INFORMATION is set as following:

/* Support SRB_FUNCTION_RESET_DEVICE */
pPCI->ResetTargetSupported = FALSE;

/* Set the supported number of Targets per bus. */
pPCI->MaximumNumberOfTargets = 1;

/* Set the supported number of LUNs per target. */
pPCI->MaximumNumberOfLogicalUnits = (UCHAR)1;

/* Set driver to run in full duplex mode */
pPCI->SynchronizationModel = StorSynchronizeFullDuplex;

/* Specify the size of SrbExtension */
pPCI->SrbExtensionSize = sizeof(vfd_wbio_ext);

/* For 64-bit systems, controller supports 64-bit addressing, */
//if (pPCI->Dma64BitAddresses == SCSI_DMA64_SYSTEM_SUPPORTED)
pPCI->Dma64BitAddresses = SCSI_DMA64_MINIPORT_FULL64BIT_SUPPORTED;

pPCI->InterruptSynchronizationMode = InterruptSynchronizePerMessage;

I don’t know why there’s a bounce buffer used.
Even if there’s a bounce buffer used, it shouldn’t change the source data.

If there is a memcpy going on then you are using a bounce buffer for dma ,
which might explain why you think you are seeing your ‘data buffer’ being
modified after completion.

Mark Roddy

On Fri, Aug 31, 2018 at 5:22 AM xxxxx@gmail.com
> wrote:
>
> > This is a storport driver. It works fine on Server 2012.
> > But the data is modified after StorPortNotification on Server 2008 R2.
> >
> > Following is the details:
> > 1. A simple test application open the block device and read first 4K
> > data. First 4k is set to ‘0x5a’ before read.
> > 2. Before StorPortNotification, we get virtual address of this srb by
> > StorPortGetSystemAddress. The 4k data in returned virtual address are all
> > ‘0x5a’.
> > 3. I set a break point when write to address (systemaddress+0x5b0). Got