Writing large disk files progressively slows

Operating system: Windows XP SP2.
File system: NTFS.

I’ve noticed that the average writing speed reduces progressively when writing to large disk files. I expect it occurs for small files also but it’s especially noticeable for large files greater than 60GB, for example.

I’ve turned the indexing service off and ensured that the file is contiguous. Changing the cluster size from 4KB to 64KB does not improve things. I’ve tried the local disk, and removable disks over USB and Firewire, and all exhibit the same problem.

Can’t find anything in the NTFS specs to suggest why the speed reduction.
Any suggestions?

Regards, Richard.

How big of a reduction are we talking about, and can you see where the data is written to physically? The outer and inner sides of an HDD have different speeds.
Especially if your drive is small (<120GB?), the speed degradation “over time” is expected, as the data is first written to the fastest than to slower and slower track.

xxxxx@compuserve.com wrote:

Operating system: Windows XP SP2.
File system: NTFS.

I’ve noticed that the average writing speed reduces progressively when writing to large disk files. I expect it occurs for small files also but it’s especially noticeable for large files greater than 60GB, for example.

I’ve turned the indexing service off and ensured that the file is contiguous. Changing the cluster size from 4KB to 64KB does not improve things. I’ve tried the local disk, and removable disks over USB and Firewire, and all exhibit the same problem.

Can’t find anything in the NTFS specs to suggest why the speed reduction.
Any suggestions?

Regards, Richard.


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@alfasp.com
To unsubscribe send a blank email to xxxxx@lists.osr.com


Kind regards, Dejan (MSN support: xxxxx@alfasp.com)
http://www.alfasp.com
File system audit, security and encryption kits.

Is it via SMB client/server pair or on the local disk?

With SMB, this is a known issue on srv.sys


Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

wrote in message news:xxxxx@ntfsd…
> Operating system: Windows XP SP2.
> File system: NTFS.
>
> I’ve noticed that the average writing speed reduces progressively when
writing to large disk files. I expect it occurs for small files also but it’s
especially noticeable for large files greater than 60GB, for example.
>
> I’ve turned the indexing service off and ensured that the file is contiguous.
Changing the cluster size from 4KB to 64KB does not improve things. I’ve tried
the local disk, and removable disks over USB and Firewire, and all exhibit the
same problem.
>
> Can’t find anything in the NTFS specs to suggest why the speed reduction.
> Any suggestions?
>
> Regards, Richard.
>

Thank you for your responses.
I’ve done more tests writing a 60GB file with random data to a LaCie 80GB USB disk.
The two variables in the tests are the write buffer size and whether the file is initially empty or has been completely written to before.

Test1: Writing 60GB to an empty file with buffer size of 2MB.
Test2: Writing 60GB to a written file with buffer size of 2MB.
Test3: Writing 60GB to an empty file with buffer size of 32KB.
Test4: Writing 60GB to a written file with buffer size of 32KB.

I’ve also tested with 16KB buffer size, but the results were identical to the 32KB case.
I’ve also tried Test3 but with first writing one 2MB buffer, to see if this would nudge the filesystem in some way, but there was no difference.

All created files had less than 5 fragments once written to, so fragmentation is not a problem.

Results listed in decreasing speed order are as follows:
Test4: Linear speed
Test2: Exponentially slowing down, 14% slower than Test4 after 60GB
Test1: Exponentially slowing down, 21% slower than Test4 after 60GB
Test3: Exponentially slowing down, 67% slower than Test4 after 60GB

The linear speed is interesting because as pointed out by Dejan you would expect the rate to slow down depending on the position on the disk, unless the density or rpm vary.

Hesitant conclusions:
It’s always faster to write to a previously written file than to an empty one.
Writing to an empty file is faster the larger the buffer size.

I have a spreadsheet or CSV available if anyone wants to see the curves.

Regards, Richard.

I see no way of test4 having linear speed unless the data is random. It’s just not physically possible, since after writing 60GB, some parts are for sure at position 20GB or
less, and some are at 60GB or more - the two positions have very distinct speeds.

However… a USB drive might not play the same as a local HDD here, since the USB protocol is probably the limiting factor and not the drive itself. Care to share specific
numbers on the write speed?

xxxxx@compuserve.com wrote:

Thank you for your responses.
I’ve done more tests writing a 60GB file with random data to a LaCie 80GB USB disk.
The two variables in the tests are the write buffer size and whether the file is initially empty or has been completely written to before.

Test1: Writing 60GB to an empty file with buffer size of 2MB.
Test2: Writing 60GB to a written file with buffer size of 2MB.
Test3: Writing 60GB to an empty file with buffer size of 32KB.
Test4: Writing 60GB to a written file with buffer size of 32KB.

I’ve also tested with 16KB buffer size, but the results were identical to the 32KB case.
I’ve also tried Test3 but with first writing one 2MB buffer, to see if this would nudge the filesystem in some way, but there was no difference.

All created files had less than 5 fragments once written to, so fragmentation is not a problem.

Results listed in decreasing speed order are as follows:
Test4: Linear speed
Test2: Exponentially slowing down, 14% slower than Test4 after 60GB
Test1: Exponentially slowing down, 21% slower than Test4 after 60GB
Test3: Exponentially slowing down, 67% slower than Test4 after 60GB

The linear speed is interesting because as pointed out by Dejan you would expect the rate to slow down depending on the position on the disk, unless the density or rpm vary.

Hesitant conclusions:
It’s always faster to write to a previously written file than to an empty one.
Writing to an empty file is faster the larger the buffer size.

I have a spreadsheet or CSV available if anyone wants to see the curves.

Regards, Richard.


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@alfasp.com
To unsubscribe send a blank email to xxxxx@lists.osr.com


Kind regards, Dejan (MSN support: xxxxx@alfasp.com)
http://www.alfasp.com
File system audit, security and encryption kits.

The 80GB LaCie USB 2.0 disk drive has rotational speed 5400 rpm and maximum interface transfer rate of 480 Mbits/sec.
The best speed was 25Mbytes/sec, the worst 15Mbytes/sec.
Below is a comma separated value table:
GB,SECS1,SECS2,SECS3,SECS4
0,0,0,0,0
1,46,45,53,40
2,92,91,108,80
3,138,136,163,121
4,185,182,218,161
5,231,228,273,202
6,279,273,327,243
7,325,319,382,283
8,372,365,440,324
9,420,411,496,365
10,468,456,552,405
11,516,501,611,446
12,564,547,669,487
13,612,592,727,528
14,659,638,785,569
15,706,683,842,609
16,754,729,902,650
17,803,774,962,692
18,850,820,1022,733
19,899,865,1084,774
20,945,910,1144,815
21,994,956,1206,856
22,1042,1002,1269,897
23,1091,1047,1331,938
24,1139,1093,1394,979
25,1186,1139,1457,1020
26,1234,1185,1519,1061
27,1281,1230,1583,1102
28,1328,1276,1647,1143
29,1376,1322,1711,1184
30,1424,1367,1776,1225
31,1472,1413,1843,1266
32,1520,1460,1909,1307
33,1568,1506,1977,1348
34,1617,1552,2044,1390
35,1664,1599,2113,1431
36,1712,1645,2181,1472
37,1761,1692,2251,1512
38,1809,1739,2323,1554
39,1858,1785,2393,1595
40,1906,1832,2464,1637
41,1955,1880,2536,1678
42,2004,1927,2609,1719
43,2052,1973,2681,1760
44,2102,2020,2755,1801
45,2150,2068,2829,1843
46,2200,2115,2904,1884
47,2250,2162,2980,1925
48,2299,2210,3058,1966
49,2349,2258,3135,2007
50,2401,2307,3213,2049
51,2452,2355,3292,2090
52,2503,2404,3370,2131
53,2555,2454,3450,2172
54,2606,2503,3530,2213
55,2660,2553,3610,2255
56,2712,2603,3691,2296
57,2763,2652,3772,2337
58,2815,2702,3853,2378
59,2867,2752,3936,2419

> The best speed was 25Mbytes/sec, the worst 15Mbytes/sec.

This is USB’s limitation and not drive’s.


Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

Curious… it’s easy to explain why the pre-allocated is faster in the same buffer size category, but not why the
32KB chunks are faster…
Are these all no-cache writes?

xxxxx@compuserve.com wrote:

The 80GB LaCie USB 2.0 disk drive has rotational speed 5400 rpm and maximum interface transfer rate of 480 Mbits/sec.
The best speed was 25Mbytes/sec, the worst 15Mbytes/sec.
Below is a comma separated value table:
GB,SECS1,SECS2,SECS3,SECS4
0,0,0,0,0
1,46,45,53,40
2,92,91,108,80
3,138,136,163,121
4,185,182,218,161
5,231,228,273,202
6,279,273,327,243
7,325,319,382,283
8,372,365,440,324
9,420,411,496,365
10,468,456,552,405
11,516,501,611,446
12,564,547,669,487
13,612,592,727,528
14,659,638,785,569
15,706,683,842,609
16,754,729,902,650
17,803,774,962,692
18,850,820,1022,733
19,899,865,1084,774
20,945,910,1144,815
21,994,956,1206,856
22,1042,1002,1269,897
23,1091,1047,1331,938
24,1139,1093,1394,979
25,1186,1139,1457,1020
26,1234,1185,1519,1061
27,1281,1230,1583,1102
28,1328,1276,1647,1143
29,1376,1322,1711,1184
30,1424,1367,1776,1225
31,1472,1413,1843,1266
32,1520,1460,1909,1307
33,1568,1506,1977,1348
34,1617,1552,2044,1390
35,1664,1599,2113,1431
36,1712,1645,2181,1472
37,1761,1692,2251,1512
38,1809,1739,2323,1554
39,1858,1785,2393,1595
40,1906,1832,2464,1637
41,1955,1880,2536,1678
42,2004,1927,2609,1719
43,2052,1973,2681,1760
44,2102,2020,2755,1801
45,2150,2068,2829,1843
46,2200,2115,2904,1884
47,2250,2162,2980,1925
48,2299,2210,3058,1966
49,2349,2258,3135,2007
50,2401,2307,3213,2049
51,2452,2355,3292,2090
52,2503,2404,3370,2131
53,2555,2454,3450,2172
54,2606,2503,3530,2213
55,2660,2553,3610,2255
56,2712,2603,3691,2296
57,2763,2652,3772,2337
58,2815,2702,3853,2378
59,2867,2752,3936,2419


Kind regards, Dejan (MSN support: xxxxx@alfasp.com)
http://www.alfasp.com
File system audit, security and encryption kits.

I’ve repeated the tests using FILE_FLAG_NO_BUFFERING and all speeds are now linear.

The conditions imposed on unbuffered writes mean that the blocksize and buffer address
have to be integer multiples of the volume sector size. Which means that in my typical
applications I’ll have to do the buffering myself which may or may not be as efficient
as writes buffered by the system.

I guess the speed variation due to the position on the physical disk is not
in evidence because of the slow USB interface.

Below is a comma separated value table with explanations of the column titles.
Regards, Richard.

Buffered:
SECS1: Writing 60GB to an empty file with buffer size of 2MB.
SECS2: Writing 60GB to a written file with buffer size of 2MB.
SECS3: Writing 60GB to an empty file with buffer size of 32KB.
SECS4: Writing 60GB to a written file with buffer size of 32KB.
Unbuffered:
SECS1u: Writing 60GB to an empty file with buffer size of 2MB.
SECS2u: Writing 60GB to a written file with buffer size of 2MB.
SECS3u: Writing 60GB to an empty file with buffer size of 32KB.
SECS4u: Writing 60GB to a written file with buffer size of 32KB.

GB,SECS1u,SECS2u,SECS3u,SECS4u,SECS1,SECS2,SECS3,SECS4
0,0,0,0,0,0,0,0,0
1,40,39,50,46,46,45,53,40
2,80,78,99,92,92,91,108,80
3,120,117,149,139,138,136,163,121
4,159,156,198,185,185,182,218,161
5,199,195,248,231,231,228,273,202
6,239,234,298,277,279,273,327,243
7,279,274,348,323,325,319,382,283
8,319,313,397,369,372,365,440,324
9,359,352,447,415,420,411,496,365
10,399,391,497,461,468,456,552,405
11,438,430,546,507,516,501,611,446
12,478,469,596,553,564,547,669,487
13,518,508,645,599,612,592,727,528
14,558,547,695,645,659,638,785,569
15,598,586,744,691,706,683,842,609
16,638,625,794,737,754,729,902,650
17,678,664,843,783,803,774,962,692
18,718,703,893,829,850,820,1022,733
19,758,742,943,874,899,865,1084,774
20,798,781,992,920,945,910,1144,815
21,838,820,1042,966,994,956,1206,856
22,878,859,1092,1012,1042,1002,1269,897
23,917,898,1141,1058,1091,1047,1331,938
24,958,937,1191,1104,1139,1093,1394,979
25,998,977,1240,1151,1186,1139,1457,1020
26,1038,1016,1290,1196,1234,1185,1519,1061
27,1078,1055,1340,1242,1281,1230,1583,1102
28,1118,1094,1389,1288,1328,1276,1647,1143
29,1157,1133,1439,1334,1376,1322,1711,1184
30,1197,1172,1489,1380,1424,1367,1776,1225
31,1237,1211,1538,1426,1472,1413,1843,1266
32,1277,1250,1588,1472,1520,1460,1909,1307
33,1317,1289,1638,1518,1568,1506,1977,1348
34,1357,1329,1688,1564,1617,1552,2044,1390
35,1397,1368,1737,1609,1664,1599,2113,1431
36,1437,1407,1787,1655,1712,1645,2181,1472
37,1477,1446,1836,1701,1761,1692,2251,1512
38,1517,1485,1886,1747,1809,1739,2323,1554
39,1557,1524,1936,1793,1858,1785,2393,1595
40,1597,1563,1986,1839,1906,1832,2464,1637
41,1637,1602,2035,1885,1955,1880,2536,1678
42,1677,1641,2085,1930,2004,1927,2609,1719
43,1718,1680,2135,1976,2052,1973,2681,1760
44,1758,1719,2185,2022,2102,2020,2755,1801
45,1798,1758,2234,2068,2150,2068,2829,1843
46,1838,1797,2284,2114,2200,2115,2904,1884
47,1878,1837,2334,2159,2250,2162,2980,1925
48,1919,1876,2383,2205,2299,2210,3058,1966
49,1959,1915,2433,2251,2349,2258,3135,2007
50,1999,1954,2483,2297,2401,2307,3213,2049
51,2039,1993,2533,2343,2452,2355,3292,2090
52,2079,2032,2582,2390,2503,2404,3370,2131
53,2120,2071,2632,2436,2555,2454,3450,2172
54,2160,2110,2682,2482,2606,2503,3530,2213
55,2200,2149,2731,2528,2660,2553,3610,2255
56,2240,2188,2781,2574,2712,2603,3691,2296
57,2281,2227,2831,2620,2763,2652,3772,2337
58,2321,2267,2880,2665,2815,2702,3853,2378
59,2361,2306,2930,2711,2867,2752,3936,2419
60,2401,2345,2980,2757,

There we are, as expected writing uncached with larger buffer is faster.
It’s a known fact that with caching there are absolutely no guarantees with regard to
write speed.

xxxxx@compuserve.com wrote:

I’ve repeated the tests using FILE_FLAG_NO_BUFFERING and all speeds are now linear.

The conditions imposed on unbuffered writes mean that the blocksize and buffer address
have to be integer multiples of the volume sector size. Which means that in my typical
applications I’ll have to do the buffering myself which may or may not be as efficient
as writes buffered by the system.

I guess the speed variation due to the position on the physical disk is not
in evidence because of the slow USB interface.

Below is a comma separated value table with explanations of the column titles.
Regards, Richard.

Buffered:
SECS1: Writing 60GB to an empty file with buffer size of 2MB.
SECS2: Writing 60GB to a written file with buffer size of 2MB.
SECS3: Writing 60GB to an empty file with buffer size of 32KB.
SECS4: Writing 60GB to a written file with buffer size of 32KB.
Unbuffered:
SECS1u: Writing 60GB to an empty file with buffer size of 2MB.
SECS2u: Writing 60GB to a written file with buffer size of 2MB.
SECS3u: Writing 60GB to an empty file with buffer size of 32KB.
SECS4u: Writing 60GB to a written file with buffer size of 32KB.

GB,SECS1u,SECS2u,SECS3u,SECS4u,SECS1,SECS2,SECS3,SECS4
0,0,0,0,0,0,0,0,0
1,40,39,50,46,46,45,53,40
2,80,78,99,92,92,91,108,80
3,120,117,149,139,138,136,163,121
4,159,156,198,185,185,182,218,161
5,199,195,248,231,231,228,273,202
6,239,234,298,277,279,273,327,243
7,279,274,348,323,325,319,382,283
8,319,313,397,369,372,365,440,324
9,359,352,447,415,420,411,496,365
10,399,391,497,461,468,456,552,405
11,438,430,546,507,516,501,611,446
12,478,469,596,553,564,547,669,487
13,518,508,645,599,612,592,727,528
14,558,547,695,645,659,638,785,569
15,598,586,744,691,706,683,842,609
16,638,625,794,737,754,729,902,650
17,678,664,843,783,803,774,962,692
18,718,703,893,829,850,820,1022,733
19,758,742,943,874,899,865,1084,774
20,798,781,992,920,945,910,1144,815
21,838,820,1042,966,994,956,1206,856
22,878,859,1092,1012,1042,1002,1269,897
23,917,898,1141,1058,1091,1047,1331,938
24,958,937,1191,1104,1139,1093,1394,979
25,998,977,1240,1151,1186,1139,1457,1020
26,1038,1016,1290,1196,1234,1185,1519,1061
27,1078,1055,1340,1242,1281,1230,1583,1102
28,1118,1094,1389,1288,1328,1276,1647,1143
29,1157,1133,1439,1334,1376,1322,1711,1184
30,1197,1172,1489,1380,1424,1367,1776,1225
31,1237,1211,1538,1426,1472,1413,1843,1266
32,1277,1250,1588,1472,1520,1460,1909,1307
33,1317,1289,1638,1518,1568,1506,1977,1348
34,1357,1329,1688,1564,1617,1552,2044,1390
35,1397,1368,1737,1609,1664,1599,2113,1431
36,1437,1407,1787,1655,1712,1645,2181,1472
37,1477,1446,1836,1701,1761,1692,2251,1512
38,1517,1485,1886,1747,1809,1739,2323,1554
39,1557,1524,1936,1793,1858,1785,2393,1595
40,1597,1563,1986,1839,1906,1832,2464,1637
41,1637,1602,2035,1885,1955,1880,2536,1678
42,1677,1641,2085,1930,2004,1927,2609,1719
43,1718,1680,2135,1976,2052,1973,2681,1760
44,1758,1719,2185,2022,2102,2020,2755,1801
45,1798,1758,2234,2068,2150,2068,2829,1843
46,1838,1797,2284,2114,2200,2115,2904,1884
47,1878,1837,2334,2159,2250,2162,2980,1925
48,1919,1876,2383,2205,2299,2210,3058,1966
49,1959,1915,2433,2251,2349,2258,3135,2007
50,1999,1954,2483,2297,2401,2307,3213,2049
51,2039,1993,2533,2343,2452,2355,3292,2090
52,2079,2032,2582,2390,2503,2404,3370,2131
53,2120,2071,2632,2436,2555,2454,3450,2172
54,2160,2110,2682,2482,2606,2503,3530,2213
55,2200,2149,2731,2528,2660,2553,3610,2255
56,2240,2188,2781,2574,2712,2603,3691,2296
57,2281,2227,2831,2620,2763,2652,3772,2337
58,2321,2267,2880,2665,2815,2702,3853,2378
59,2361,2306,2930,2711,2867,2752,3936,2419
60,2401,2345,2980,2757,


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@alfasp.com
To unsubscribe send a blank email to xxxxx@lists.osr.com


Kind regards, Dejan (MSN support: xxxxx@alfasp.com)
http://www.alfasp.com
File system audit, security and encryption kits.

Not to mention that uncached writes also avoid the highly unpleasant side effect of paging the entire machine out to make room for the file system cache, thus making the system all but unusable (as the OP noticed).

  • S

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Dejan Maksimovic
Sent: Monday, August 04, 2008 2:44 PM
To: Windows File Systems Devs Interest List
Subject: Re: [ntfsd] Writing large disk files progressively slows

There we are, as expected writing uncached with larger buffer is faster.
It’s a known fact that with caching there are absolutely no guarantees with regard to
write speed.

xxxxx@compuserve.com wrote:

I’ve repeated the tests using FILE_FLAG_NO_BUFFERING and all speeds are now linear.

The conditions imposed on unbuffered writes mean that the blocksize and buffer address
have to be integer multiples of the volume sector size. Which means that in my typical
applications I’ll have to do the buffering myself which may or may not be as efficient
as writes buffered by the system.

I guess the speed variation due to the position on the physical disk is not
in evidence because of the slow USB interface.

Below is a comma separated value table with explanations of the column titles.
Regards, Richard.

Buffered:
SECS1: Writing 60GB to an empty file with buffer size of 2MB.
SECS2: Writing 60GB to a written file with buffer size of 2MB.
SECS3: Writing 60GB to an empty file with buffer size of 32KB.
SECS4: Writing 60GB to a written file with buffer size of 32KB.
Unbuffered:
SECS1u: Writing 60GB to an empty file with buffer size of 2MB.
SECS2u: Writing 60GB to a written file with buffer size of 2MB.
SECS3u: Writing 60GB to an empty file with buffer size of 32KB.
SECS4u: Writing 60GB to a written file with buffer size of 32KB.

GB,SECS1u,SECS2u,SECS3u,SECS4u,SECS1,SECS2,SECS3,SECS4
0,0,0,0,0,0,0,0,0
1,40,39,50,46,46,45,53,40
2,80,78,99,92,92,91,108,80
3,120,117,149,139,138,136,163,121
4,159,156,198,185,185,182,218,161
5,199,195,248,231,231,228,273,202
6,239,234,298,277,279,273,327,243
7,279,274,348,323,325,319,382,283
8,319,313,397,369,372,365,440,324
9,359,352,447,415,420,411,496,365
10,399,391,497,461,468,456,552,405
11,438,430,546,507,516,501,611,446
12,478,469,596,553,564,547,669,487
13,518,508,645,599,612,592,727,528
14,558,547,695,645,659,638,785,569
15,598,586,744,691,706,683,842,609
16,638,625,794,737,754,729,902,650
17,678,664,843,783,803,774,962,692
18,718,703,893,829,850,820,1022,733
19,758,742,943,874,899,865,1084,774
20,798,781,992,920,945,910,1144,815
21,838,820,1042,966,994,956,1206,856
22,878,859,1092,1012,1042,1002,1269,897
23,917,898,1141,1058,1091,1047,1331,938
24,958,937,1191,1104,1139,1093,1394,979
25,998,977,1240,1151,1186,1139,1457,1020
26,1038,1016,1290,1196,1234,1185,1519,1061
27,1078,1055,1340,1242,1281,1230,1583,1102
28,1118,1094,1389,1288,1328,1276,1647,1143
29,1157,1133,1439,1334,1376,1322,1711,1184
30,1197,1172,1489,1380,1424,1367,1776,1225
31,1237,1211,1538,1426,1472,1413,1843,1266
32,1277,1250,1588,1472,1520,1460,1909,1307
33,1317,1289,1638,1518,1568,1506,1977,1348
34,1357,1329,1688,1564,1617,1552,2044,1390
35,1397,1368,1737,1609,1664,1599,2113,1431
36,1437,1407,1787,1655,1712,1645,2181,1472
37,1477,1446,1836,1701,1761,1692,2251,1512
38,1517,1485,1886,1747,1809,1739,2323,1554
39,1557,1524,1936,1793,1858,1785,2393,1595
40,1597,1563,1986,1839,1906,1832,2464,1637
41,1637,1602,2035,1885,1955,1880,2536,1678
42,1677,1641,2085,1930,2004,1927,2609,1719
43,1718,1680,2135,1976,2052,1973,2681,1760
44,1758,1719,2185,2022,2102,2020,2755,1801
45,1798,1758,2234,2068,2150,2068,2829,1843
46,1838,1797,2284,2114,2200,2115,2904,1884
47,1878,1837,2334,2159,2250,2162,2980,1925
48,1919,1876,2383,2205,2299,2210,3058,1966
49,1959,1915,2433,2251,2349,2258,3135,2007
50,1999,1954,2483,2297,2401,2307,3213,2049
51,2039,1993,2533,2343,2452,2355,3292,2090
52,2079,2032,2582,2390,2503,2404,3370,2131
53,2120,2071,2632,2436,2555,2454,3450,2172
54,2160,2110,2682,2482,2606,2503,3530,2213
55,2200,2149,2731,2528,2660,2553,3610,2255
56,2240,2188,2781,2574,2712,2603,3691,2296
57,2281,2227,2831,2620,2763,2652,3772,2337
58,2321,2267,2880,2665,2815,2702,3853,2378
59,2361,2306,2930,2711,2867,2752,3936,2419
60,2401,2345,2980,2757,


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@alfasp.com
To unsubscribe send a blank email to xxxxx@lists.osr.com


Kind regards, Dejan (MSN support: xxxxx@alfasp.com)
http://www.alfasp.com
File system audit, security and encryption kits.

Maybe write throttling can cause such performance degradation. There is a global threshold in Cache Manager (CCDirtyPageTreshold). If number of dirty pages in cache reaches this threshold, all writing threads are blocked, until lazy writer releases pages for usage. Use !defwrite command to determine if write throttling is in effect. The writing thread is blocked in CcCanIWrite() function in such situation. By bigger buffer in cached write you only influence how fast data reaches the cache but CC determines the size of paged non-cached writes, which are usually like 64kB. It seems that your data aggressively consume whole cache and affect system performance (also consume physical pages which are needed somewhere else). You can set dirty page threshold for specific file by CcSetDirtyPageThreshold() which might prevent from such aggressive behavior. Unfortunately it can be called only from driver (more likely driver on the stack). For experimental purpose you can use debugger for its setting. It is located in _FILE_OBJECT.SectionObjectPointer.SharedCacheMap and it is in pages.

-bg

kd> dt nt!_FILE_OBJECT
+0x000 Type : Int2B
+0x002 Size : Int2B
+0x004 DeviceObject : Ptr32 _DEVICE_OBJECT
+0x008 Vpb : Ptr32 _VPB
+0x00c FsContext : Ptr32 Void
+0x010 FsContext2 : Ptr32 Void
+0x014 SectionObjectPointer : Ptr32 _SECTION_OBJECT_POINTERS

kd> dt nt!_SECTION_OBJECT_POINTERS
+0x000 DataSectionObject : Ptr32 Void
+0x004 SharedCacheMap : Ptr32 Void
+0x008 ImageSectionObject : Ptr32 Void

dt nt!_SHARED_CACHE_MAP
+0x000 NodeTypeCode : Int2B
+0x002 NodeByteSize : Int2B
… …
+0x0a8 DirtyPageThreshold : Uint4B
+0x0ac LazyWritePassCount : Uint4B

> It’s a known fact that with caching there are absolutely no guarantees
with

regard to
write speed.

Cache exists a) to just allow non-sector-aligned IO, this is not optimization,
but just the feature itself - traditional fwrite() and write() cannot be
implemented without caching b) to optimize the pattern of “many reads/writes to
a small file range” c) to optimize the pattern of “reopen and re-access the
same file many times”.

As about large linear writes like huge file copies or sequential reads of the
whole file - caching is often not an optimization there, but a bottleneck. The
reason is that with such access patterns you rely on Cc’s read-ahead and
lazy-write features, which are no so customizable compared to your own code for
them (64K portions only etc), and can become even a larger bottleneck due to
consuming lots of RAM.

The main win from caching - reusing the data already in-RAM for subsequent
accesses - does not exist on a large file copy scenario, when each data portion
is accessed only once.

Caching with SMB redir/server is yet another can of worms performance-wise.


Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

Maxim S. Shatskih wrote:

Cache exists a) to just allow non-sector-aligned IO, this is not optimization,
but just the feature itself - traditional fwrite() and write() cannot be
implemented without caching b) to optimize the pattern of “many reads/writes to
a small file range” c) to optimize the pattern of “reopen and re-access the
same file many times”.

There was no need for pattern of “copy a huge .iso file over network”
when all this stuff was designed ? :slight_smile:

> There was no need for pattern of “copy a huge .iso file over network”

when all this stuff was designed ? :slight_smile:

On most real-world patterns - except the huge file copies or huge file
sequential reads or huge file generation - cache introduces a loss and not a
win.

Nevertheless, cache is always a win in most file access patterns like reading
.DOC/.XLS, or reading the XML config file or such. Some simpler database
engines (Postgres) also use record-wise read()/write() calls and thus use the
kernel’s FS’s cache a lot.

Probably MS’s Jet (from Access, not ESE.DLL which is more advanced and is
targeted for more serious things) is also such, so are legacy ISAMs like
Foxpro’s.

Now note that noncache mode has a major limitation of alignment, which makes it
unsuitable for most common tasks.

So, the default in Windows and UNIXen is - cached file, it is free from this
limitation and is recommended for standard general-purpose file IO.

Noncached IO - due to its limitations partly - is considered to be “advanced”
both in Windows and in UNIXen. It is suggested that
O_DIRECT/FILE_FLAG_NO_BUFFERING is only used if the developer really knows what
he does.

But yes, for apps like database ISAMs with their own cache (MSSQLServer or ESE)
of for huge file copies - cache is more like a loss.


Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

When it was designed huge ISO files couldn’t fit on the largest hard disk drive
available commercially :wink:

“Pavel A.” wrote:

Maxim S. Shatskih wrote:
> Cache exists a) to just allow non-sector-aligned IO, this is not optimization,
> but just the feature itself - traditional fwrite() and write() cannot be
> implemented without caching b) to optimize the pattern of “many reads/writes to
> a small file range” c) to optimize the pattern of “reopen and re-access the
> same file many times”.

There was no need for pattern of “copy a huge .iso file over network”
when all this stuff was designed ? :slight_smile:


Kind regards, Dejan (MSN support: xxxxx@alfasp.com)
http://www.alfasp.com
File system audit, security and encryption kits.

Thanks everybody for your input.
Regards, Richard.