switch off the buffer io

Hi, All,

Recently, we try to test our xHCI host.

Our xHCI host is a two port host, when test, we plug two Superspeed udisk on each port.
So we call them as SS1 and SS2.
Our system is Win8.1 64 bit and with 8GB DDR memory.

We found a case, when copy a 2GB file from the Udisk, the first time, it will take normal time, but the second and later loop, the file copy finished very soon, but this is not what we need, we need the file copy each time is from udisk to PC harddrive.
So I think it is because it is cause by the buffered IO feature of Windows OS.

The file copy action is from our own script: with COPY command.
And we check that the windows command XCOPY with /j option, this will close the buffer IO, so each time the file copy could from udisk instead of buffered data in DDR memory.
But it seems that the XCOPY /j option not actually switch off the buffered IO case.

Any one have experience on how to switch this buffer io feature?

On Oct 18, 2014, at 6:39 AM, workingmailing@163.com wrote:

Recently, we try to test our xHCI host.

Our xHCI host is a two port host, when test, we plug two Superspeed udisk on each port.
So we call them as SS1 and SS2.
Our system is Win8.1 64 bit and with 8GB DDR memory.

We found a case, when copy a 2GB file from the Udisk, the first time, it will take normal time, but the second and later loop, the file copy finished very soon, but this is not what we need, we need the file copy each time is from udisk to PC harddrive.
So I think it is because it is cause by the buffered IO feature of Windows OS.

Why? For performance testing? If so, then just copy a different file each time. It?s hardly reasonable to ask the operating system to disable an important optimization just so you can gather statistics.

Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Thank you Tim.

The test designer original testing aim is to copy from superspeed u disk to PC hard driver looply, each time is from u disk, but currently, only the first case is, the second and later is from DDR memory, then the copy speed is much quick.

In this case, copy once and copy looply more than once, is without difference.

Why do you think that xcopy /j is not working? The documentation states that this will do non-cached I/O and I suspect that it’s just passing the appropriate flag to CopyFileEx.

Of course, if you’re trying to test specific behavior you can always write your own test app.

-scott
OSR
@OSRDrivers

See if unmounting the fs and remounting it makes a difference?

First operation might result in setting parameters that might improve
performance for sequential I/O. Also you might just want to try things with
rawfs + iometer, if you haven’t. As you might have observed running a copy
command can have variable results :slight_smile:

Write your own copy tool, it is trivial.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

wrote in message news:xxxxx@ntdev…
> Hi, All,
>
> Recently, we try to test our xHCI host.
>
> Our xHCI host is a two port host, when test, we plug two Superspeed udisk on each port.
> So we call them as SS1 and SS2.
> Our system is Win8.1 64 bit and with 8GB DDR memory.
>
> We found a case, when copy a 2GB file from the Udisk, the first time, it will take normal time, but the second and later loop, the file copy finished very soon, but this is not what we need, we need the file copy each time is from udisk to PC harddrive.
> So I think it is because it is cause by the buffered IO feature of Windows OS.
>
> The file copy action is from our own script: with COPY command.
> And we check that the windows command XCOPY with /j option, this will close the buffer IO, so each time the file copy could from udisk instead of buffered data in DDR memory.
> But it seems that the XCOPY /j option not actually switch off the buffered IO case.
>
> Any one have experience on how to switch this buffer io feature?
>
>

To Scott,

when i change the batch file from COPY to XCOPY /j, the effect is same, namely the second and later is very quickly.

I re-considering the /j option:
there are three IOs in Windows, they are:
buffered, direct IO and neither.
Here /j option switch off the buffer IO case, I guess, it change to the more fast one: direct IO or neither, these two is more effective than buffer IO because it omit the copy to and copy from the temped buffer.

If /j is used as this, I think it is not the method to fix my problem.

We can use things like HD_SPEED.

The script is from certification organization.
In the pervious days, the designated test system is not 8GB, 64bit Win8.1 OS, so this file copy not from udisk but from DDR case is not appeared.
But now after organization require the update, this issue appear.

We need to use the organizer supplied batch file for testing, because when certificaiton, the script is used.
So this is why I have to modify the batch file, because we have to keep the environment more likely the same as certification procedure.

> buffered, direct IO and neither

These are not types of I/O. These are types of buffer passing between user mode and kernel mode. They are unrelated to the file caching.

If you want to disable file caching, write your own test program that opens the file with FILE_FLAG_NO_BUFFERING and reads it, preferably with large I/O sizes.

> I re-considering the /j option:

there are three IOs in Windows, they are:
buffered, direct IO and neither.
Here /j option switch off the buffer IO case

No. /j switches off caching using FILE_NO_INTERMEDIATE_BUFFERING

3 IO kinds are about driver-to-kernel interface, not about FSD file modes.

FSD always uses neither IO, BTW :slight_smile:


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

thank you all.

I draft a console, with the CopyFileEx, which use the FILE_FLAG_NO_BUFFERING flag.
It get the requirement of: every time copy file from udisk to PC’s harddrive, instead of the second and later time from the DDR memory.
And from the CopyFileEx callback debug print, it is called every 64KB is transferred.

But a new requirement is asked by others:

  1. file compare/data integraty check between the source file and the destination.
    It seems that this compare should take after the file totally copied?

  2. If in between the file copy, the link has some issue(may be cause by xHCI host, SS hub, or device), which cause some USB packet data error, we need the console print some special character, or more better, generate some special USB pattern packet, so the USB analyzer could trigger by this special USB pattern packet.
    Here the hard point is, how can I detect the transfer error in between, or how can I know the error as quickly as I can, when the error happened, or when the source data is differ from what I readed data.

Requirement 1 is coarse, only when totally file transfer is done, we get to known there is something wrong in the file copy, for a large file copy, USB analyzer have not enough buffer to archive the trace.

Where requriement 2 is more useful for USB link error detect, report, and for engineer analyzing when error happened, the USB analyzer is triggered by the special pattern.
We can set a usb analyzer buffer big enough to save the trace only before and after the error point.

  1. Here I use the CopyFileEx.
    If use the CreateFile, ReadFile and WriteFile, how can the triple functions finish a very large file copy, as you known, the readfile and writefile with the bufferlength is limited, definitely not as much as 2GB or more.
    Any example for it?

>If use the CreateFile, ReadFile and WriteFile, how can the triple functions finish a very large file copy

This question doesn’t make sense. Do you know you can use loops in a C program? while() {} ?

> Do you know you can use loops in a C program? while() {} ?

:-))))))))))))))))) you made my day :slight_smile:


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

workingmailing@163.com wrote:

But a new requirement is asked by others:

  1. file compare/data integraty check between the source file and the destination.
    It seems that this compare should take after the file totally copied?

Such an integrity check hardly seems necessary, but yes, I would wait
until the file is totally copied. Otherwise, you lose the performance
benefits of long sequential operations.

  1. If in between the file copy, the link has some issue(may be cause by xHCI host, SS hub, or device), which cause some USB packet data error, we need the console print some special character, or more better, generate some special USB pattern packet, so the USB analyzer could trigger by this special USB pattern packet.
    Here the hard point is, how can I detect the transfer error in between, or how can I know the error as quickly as I can, when the error happened, or when the source data is differ from what I readed data.

You can’t. USB handles all of that automatically. All packets have a
checksum. If a packet is damaged, the hardware will retry the operation
until it succeeds. The only way you would know is by looking for
retries in a hardware USB analyzer. A software analyzer is not good enough.

Where requriement 2 is more useful for USB link error detect, report, and for engineer analyzing when error happened, the USB analyzer is triggered by the special pattern.
We can set a usb analyzer buffer big enough to save the trace only before and after the error point.

You need to sit down and figure out exactly what kind of errors you
expect to encounter. USB errors on the wire are all handled
automatically, without intervention, and without notice. You’ll never
know about them. If the file doesn’t compare after the transfer is
complete, that means your device screwed up the data after it got
there. That’s not a protocol problem, so a USB trace won’t help.

  1. Here I use the CopyFileEx.
    If use the CreateFile, ReadFile and WriteFile, how can the triple functions finish a very large file copy, as you known, the readfile and writefile with the bufferlength is limited, definitely not as much as 2GB or more.
    Any example for it?

while( 1 )
{
ReadFile();
if( no data was transferred [meaning end of file] )
break;
WriteFile();
}


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Thank you Tim.

Does Windows have any API for two files compare?

I rethink it:
Readfile read some data(partial of the file) from U disk, and Writefile to a file located in PC’s SATA HDD/SSD drive.
Or the other direction, read some data from PC’s sata HDD/SSD drive, and write to u disk.

In this circumstance, I can only check data integrity after total file copy is finished.
Then you can compare the file in udisk, and file in sata drive.
Otherwise, you read partial of the file into a buffer(DDR memory), for data integrity, you need another “correct” data for compare with the data in the readed buffer, but there no correct data here.

My original design is:
the readed buffer compare with a “correct data”, if it is not same, then I send a special USB packet, for analyzer to trig.
But it seems, this can not implemented, cause no “correct data” exist.

My question about createfile, readfile, writefile to implement the file copy is:
does the 2nd read and 2nd write is after the first read/write,
for example, first read/write from file offset 0 - 64K
and does the second read/write is from file offset 64K to 128K?
Because I can’t file the offset parameter in readfile and writefile API.

>Does Windows have any API for two files compare?

RtlCompareMemory?

workingmailing@163.com wrote:

Does Windows have any API for two files compare?

No, because there’s no obvious “one way” for such an API to work. Do
you just return pass/fail? Do you return the location of the first
mismatch? Do you return all mismatches?

Fortunately, it’s trivial to write your own.

I rethink it:
Readfile read some data(partial of the file) from U disk, and Writefile to a file located in PC’s SATA HDD/SSD drive.
Or the other direction, read some data from PC’s sata HDD/SSD drive, and write to u disk.

In this circumstance, I can only check data integrity after total file copy is finished.
Then you can compare the file in udisk, and file in sata drive.
Otherwise, you read partial of the file into a buffer(DDR memory), for data integrity, you need another “correct” data for compare with the data in the readed buffer, but there no correct data here.

My original design is:
the readed buffer compare with a “correct data”, if it is not same, then I send a special USB packet, for analyzer to trig.
But it seems, this can not implemented, cause no “correct data” exist.

I cannot believe how complicated you are making this. You are thinking
about this problem WAY too hard.

Please take a step back, and ask yourself “What is the problem I am
trying to solve here?” If you are trying to copy a file and verify that
the file was copied correctly, then you just answered the question on
how to do the copy.

// Copy

CreateFile( hSource, GENERIC_READ, … );
CreateFile( hDest, GENERIC_WRITE, … );
std::vector buffer1(BUFFERSIZE);
while( forever )
{
DWORD dwActual;
ReadFile( hSource, &buffer1[0], buffer1.size(), &dwActual,
nullptr );
if( !dwActualBytes )
break;
WriteFile( hDest, &buffer1[0], dwActual, &dwActual, nulltpr );
}
CloseHandle( hDest );

// Compare

SetFilePointer( hSource, 0, nullptr, FILE_BEGIN );
CreateFile( hDest, GENERIC_READ, … );
std::vector buffer2(BUFFERSIZE);
int Position = 0;
while( forever )
{
DWORD dwActual1, dwActual2;
ReadFile( hSource, &buffer1[0], buffer1.size(), &dwActual1,
nullptr );
ReadFile( hDest, &buffer2[0], buffer2.size(), &dwActual2, nullptr );
if( dwActual1 != dwActual2 )
{
printf( “One file ended early, read %d and %d at %d\n”,
dwActual1, dwActual2, Position );
break;
}
if( memcmp( &buffer1[0], &buffer2[0], dwActual1 ) != 0 )
{
printf( “Files differ in block at %d\n”, Position );
break;
}
Position += dwactual1;
}
CloseHandle( hSource );
CloseHandle( hDest );

If you actually want to do your comparing while you are doing the copy,
you can also do that, by opening the destination file as
GENERIC_READ|GENERIC_WRITE and using SetFilePointer, but I think that’s
a stupid option, and it will reduce your performance significantly.

int Position = 0;
while( forever )
{
DWORD dwActual1, dwActual2;
ReadFile( hSource, &buffer1[0], buffer1.size(), &dwActual1,
nullptr );
WriteFile( hDest, &buffer1[0], &dwActual1, &dwActual2, nullptr );
SetFilePointer( hDest, Position, nullptr, FILE_BEGIN );
ReadFile( hDest, &buffer2[0], buffer2.size(), &dwActual2, nullptr );
if( dwActual1 != dwActual2 )
{
printf( “One file ended early, read %d and %d at %d\n”,
dwActual1, dwActual2, Position );
break;
}
if( memcmp( &buffer1[0], &buffer2[0], dwActual1 ) != 0 )
{
printf( “Files differ in block at %d\n”, Position );
break;
}
Position += dwactual1;
}

See? It’s just not that hard. You simply have to think about your
problem in words, and turn the words into code.

> My question about createfile, readfile, writefile to implement the file copy is:
> does the 2nd read and 2nd write is after the first read/write,
> for example, first read/write from file offset 0 - 64K
> and does the second read/write is from file offset 64K to 128K?
> Because I can’t file the offset parameter in readfile and writefile API.

The file system stores a current offset with your file handle. Every
time you read or write, it advances that offset. You can use
SetFilePointer to get or change that offset


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

I?d like to remind folks some files systems, like NTFS, have multiple data streams, so it?s a bit more complex to copy/compare a file than a read/write loop of the primary stream. You also might want to duplicate attributes, like the dates, so the copy matches the original. You probably need to do something appropriate if the ?file? is actually a reparse point. And then there are sparse files, which may be ugly if you don?t also make the destination sparse.

The days of a file always just being a simple sequence of bytes is long gone.

Jan

workingmailing@163.commailto:workingmailing wrote:

Does Windows have any API for two files compare?

No, because there’s no obvious “one way” for such an API to work. Do you just return pass/fail? Do you return the location of the first mismatch? Do you return all mismatches?

Fortunately, it’s trivial to write your own.</mailto:workingmailing>

thank you Tim.

Our hardware engineer find the root cause in their hardware.

It is when xHCI host fetch data from the host DDR memory, the xHCI host fetch the data from wrong place(ddr memory address) sometimes (not all the times).
In the following, no bug happen in the USB link.

Testing integrity using a filesystem on the target is not encouraged because an unreliable link could damage the filesystem itself and not just the file. You wouldn’t then be able to copy any file. You may not be able to format anyway. They may even be cases where the USB key itself could be broken. So you must use raw disk access to test integrity (read/write to sectors directly).

You should have worked on integrity first because integrity ensures that the I/O path is reliable. Performance should come second in the testing process.

Regarding data tranfers, integrity is checked using checksums or hashes. Hashes make the best choice because the recent algorithms produce less collisions (different files having the same hashe).

The procedure is simple:

  1. On the host, create a large file with random data and whose size is a multiple of the target disk’s sector size…
  2. Calculate the hashe of the file with the sha256 algorithm (sufficiant).
  3. Write the file to the target disk as a raw write operation.
  4. Read back the sectors on the target and save the content on the host.
  5. Calcule the hashe of the file generated by the previous read ooeration.
  6. Compare the two hashes: If they matche, the transfer was ok, otherwise, the link is not reliable.

Generating files with random data should be easy from the Win32 BCRYPT API. Look at the CryptGenRandom API:

http://msdn.microsoft.com/en-us/library/windows/desktop/aa379942(v=vs.85).aspx

A Technet developper provides the “PowerShell File Checksum Integrity Verifier” for file hashing, a really good job:

https://gallery.technet.microsoft.com/PowerShell-File-Checksum-e57dcd67

Thank you Geek for more information.

Actually, at first, I write a USB application base on the USBSamp driver.

It is generate the
writefile with: CBW(31 bytes, BOT write command),
writefile with: Data stage(Write data),
readfile with: CSW(13 bytes, status)

then
writefile with: CBW(31 bytes, BOT read command),
readfile with: data stage(read data),
readfile with: csw(13 byte, status stage)

because this have no file system in the data transfer loop.
Just the App, usbsamp driver, and host driver.
And it could check the data integrity.

But here, for USB host check, USB-IF use the batch file, copy and comp(compare file) for data integrity check.
Therefore, I was asked to not use the app for test, and develop a new console with the CopyFileEx.