Force Flushing FS MetaData

NTFSD
1

Hello all,

is there a way to force flush the NTFS metadata to the disk at a particular
time? I have tried *ZwFlushBuffersFile* but that doesnt seem to work even
though it returns success.

Here is the scenario:

  1. i do a CreateFileW(pwcszFileName, GENERIC_READ | GENERIC_WRITE,
    FILE_SHARE_READ
    | FILE_SHARE_WRITE, 0, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL |
    FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH, 0);
  2. I set file size by SetFilePointerEx( hFile,liDistanceToMove,NULL,
    FILE_BEGIN);
  3. set the EOF SetEndOfFile(hFile);
  4. zwFlushBuffersFile(hFile,&IoStatusBlock);
  5. CloseHandle(hFile)

right after this, when I look at the MFT to see if the entry has been
created, it doesn’t show it (we have debug code to parse NTFS correctly)

So is there a way via some IOCTL or otherwise to ask NTFS to flushthe FS
meta data.

Would IOCTL_VOLSNAP_FLUSH_AND_HOLD_WRITES work?

thanks

amit

–

  • amitr0
2

As has been said many times in the past there is no way to force NTFS to
flush any data (either meta or file) in a manner that can be relied on,
except for dismounting the volume.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

From: amitr0 [mailto:xxxxx@gmail.com]
Posted At: Monday, June 21, 2010 10:19 AM
Posted To: ntfsd
Conversation: Force Flushing FS MetaData
Subject: Force Flushing FS MetaData

Hello all,
 
is there a way to force flush the NTFS metadata to the disk at a
particular time? I have tried ZwFlushBuffersFile but that doesnt seem to
work even though it returns success.
 
Here is the scenario:
 

  1. i do a CreateFileW(pwcszFileName, GENERIC_READ | GENERIC_WRITE,
    FILE_SHARE_READ | FILE_SHARE_WRITE, 0, CREATE_ALWAYS,
    FILE_ATTRIBUTE_NORMAL | FILE_FLAG_NO_BUFFERING |
    FILE_FLAG_WRITE_THROUGH, 0);
  2. I set file size by SetFilePointerEx( hFile,liDistanceToMove,NULL,
    FILE_BEGIN);
  3. set the EOF SetEndOfFile(hFile);
  4. zwFlushBuffersFile(hFile,&IoStatusBlock);
  5. CloseHandle(hFile)
     
    right after this, when I look at the MFT to see if the entry has been
    created, it doesn’t show it (we have debug code to parse NTFS correctly)
     
    So is there a way via some IOCTL or otherwise to ask NTFS to flushthe FS
    meta data.
     
     
    Would IOCTL_VOLSNAP_FLUSH_AND_HOLD_WRITES work?
     
    thanks
     
    amit
     

–

  • amitr0

__________ Information from ESET Smart Security, version of virus
signature database 5215 (20100621) __________

The message was checked by ESET Smart Security.

http://www.eset.com

3

One disk drive vendor told me (and this is why you should never let =
techies
onto the show floor): “When you flush data to the drive, it is flushed =
to
the onboard cache of the drive, and the driver declares that the data =
has
been successfully flushed. At some future time, this cached data is =
written
to the platter. We have no way to let you guarantee that the data has =
been
flushed to the platter.” So, I ask, “So if there is a power failure =
while
the data is in the drive’s onboard cache, you can’t guarantee the =
integrity
of the data. This would be particularly bad if I’m using a transacted
database. Right?” Reply: “Yes, but then we just blame it on Microsoft”.
joe

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Monday, June 21, 2010 10:22 AM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] Force Flushing FS MetaData

As has been said many times in the past there is no way to force NTFS to
flush any data (either meta or file) in a manner that can be relied on,
except for dismounting the volume.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

From: amitr0 [mailto:xxxxx@gmail.com]
Posted At: Monday, June 21, 2010 10:19 AM
Posted To: ntfsd
Conversation: Force Flushing FS MetaData
Subject: Force Flushing FS MetaData

Hello all,
=A0
is there a way to force flush the NTFS metadata to the disk at a
particular time? I have tried ZwFlushBuffersFile but that doesnt seem to
work even though it returns success.
=A0
Here is the scenario:
=A0

  1. i do a CreateFileW(pwcszFileName, GENERIC_READ | GENERIC_WRITE,
    FILE_SHARE_READ | FILE_SHARE_WRITE, 0, CREATE_ALWAYS,
    FILE_ATTRIBUTE_NORMAL | FILE_FLAG_NO_BUFFERING |
    FILE_FLAG_WRITE_THROUGH, 0);
  2. I set file size by SetFilePointerEx( hFile,liDistanceToMove,NULL,
    FILE_BEGIN);
  3. set the EOF SetEndOfFile(hFile);
  4. zwFlushBuffersFile(hFile,&IoStatusBlock);
  5. CloseHandle(hFile)
    =A0
    right after this, when I look at the MFT to see if the entry has been
    created, it doesn’t show it (we have debug code to parse NTFS correctly)
    =A0
    So is there a way via some IOCTL or otherwise to ask NTFS to flushthe FS
    meta data.
    =A0
    =A0
    Would IOCTL_VOLSNAP_FLUSH_AND_HOLD_WRITES work?
    =A0
    thanks
    =A0
    amit
    =A0

–

  • amitr0

__________ Information from ESET Smart Security, version of virus
signature database 5215 (20100621) __________

The message was checked by ESET Smart Security.

http://www.eset.com

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:=20
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=3DListServer

–=20
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

4

Actually, those of us who track fault tolerance topics have found it is
much worse than that. First they are now putting operating systems on
some drives so they can have more intelligence, and guess what the OS’es
crash every so often. Second they are caching the data in high density
DRAM without ECC, and a Google published last year the failure rates on
this is somewhere between 1000 and 10000 times the data published by the
memory chip companies. Finally, the push for denser and denser drives,
has upped the error rates on accessing the drives is going up faster
than what many vendors are publishing.

Bottom line is the reliability of data stored on a hard drive (without
raid and a lot of backups) has decreased significantly in the last 10
years.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: Joseph M. Newcomer [mailto:xxxxx@flounder.com]
Posted At: Monday, June 21, 2010 12:40 PM
Posted To: ntfsd
Conversation: Force Flushing FS MetaData
Subject: RE: Force Flushing FS MetaData

One disk drive vendor told me (and this is why you should never let
techies
onto the show floor): “When you flush data to the drive, it is flushed
to the
onboard cache of the drive, and the driver declares that the data has
been
successfully flushed. At some future time, this cached data is
written to the
platter. We have no way to let you guarantee that the data has been
flushed
to the platter.” So, I ask, “So if there is a power failure while the
data is
in the drive’s onboard cache, you can’t guarantee the integrity of the
data.
This would be particularly bad if I’m using a transacted database.
Right?”
Reply: “Yes, but then we just blame it on Microsoft”.
joe

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Monday, June 21, 2010 10:22 AM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] Force Flushing FS MetaData

As has been said many times in the past there is no way to force NTFS
to flush
any data (either meta or file) in a manner that can be relied on,
except for
dismounting the volume.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

From: amitr0 [mailto:xxxxx@gmail.com] Posted At: Monday, June 21,
2010
10:19 AM Posted To: ntfsd
Conversation: Force Flushing FS MetaData
Subject: Force Flushing FS MetaData

Hello all,

is there a way to force flush the NTFS metadata to the disk at a
particular
time? I have tried ZwFlushBuffersFile but that doesnt seem to work
even though
it returns success.

Here is the scenario:

  1. i do a CreateFileW(pwcszFileName, GENERIC_READ | GENERIC_WRITE,
    FILE_SHARE_READ | FILE_SHARE_WRITE, 0, CREATE_ALWAYS,
    FILE_ATTRIBUTE_NORMAL |
    FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH, 0); 2. I set file
    size by
    SetFilePointerEx( hFile,liDistanceToMove,NULL, FILE_BEGIN); 3. set the
    EOF
    SetEndOfFile(hFile); 4. zwFlushBuffersFile(hFile,&IoStatusBlock);
  2. CloseHandle(hFile)

right after this, when I look at the MFT to see if the entry has been
created,
it doesn’t show it (we have debug code to parse NTFS correctly)

So is there a way via some IOCTL or otherwise to ask NTFS to flushthe
FS meta
data.

Would IOCTL_VOLSNAP_FLUSH_AND_HOLD_WRITES work?

thanks

amit

–

  • amitr0

__________ Information from ESET Smart Security, version of virus
signature
database 5215 (20100621) __________

The message was checked by ESET Smart Security.

http://www.eset.com

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars (including our
new fs
mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

–
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.

__________ Information from ESET Smart Security, version of virus
signature
database 5215 (20100621) __________

The message was checked by ESET Smart Security.

http://www.eset.com

5

If you could dismount the volume, it would do a flush.

–Mark Cariddi
OSR, Open Systems Resources, Inc

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Monday, June 21, 2010 12:51 PM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] Force Flushing FS MetaData

Actually, those of us who track fault tolerance topics have found it is
much worse than that. First they are now putting operating systems on
some drives so they can have more intelligence, and guess what the OS’es
crash every so often. Second they are caching the data in high density
DRAM without ECC, and a Google published last year the failure rates on
this is somewhere between 1000 and 10000 times the data published by the
memory chip companies. Finally, the push for denser and denser drives,
has upped the error rates on accessing the drives is going up faster
than what many vendors are publishing.

Bottom line is the reliability of data stored on a hard drive (without
raid and a lot of backups) has decreased significantly in the last 10
years.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: Joseph M. Newcomer [mailto:xxxxx@flounder.com]
Posted At: Monday, June 21, 2010 12:40 PM
Posted To: ntfsd
Conversation: Force Flushing FS MetaData
Subject: RE: Force Flushing FS MetaData

One disk drive vendor told me (and this is why you should never let
techies
onto the show floor): “When you flush data to the drive, it is flushed
to the
onboard cache of the drive, and the driver declares that the data has
been
successfully flushed. At some future time, this cached data is
written to the
platter. We have no way to let you guarantee that the data has been
flushed
to the platter.” So, I ask, “So if there is a power failure while the
data is
in the drive’s onboard cache, you can’t guarantee the integrity of the
data.
This would be particularly bad if I’m using a transacted database.
Right?”
Reply: “Yes, but then we just blame it on Microsoft”.
joe

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Monday, June 21, 2010 10:22 AM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] Force Flushing FS MetaData

As has been said many times in the past there is no way to force NTFS
to flush
any data (either meta or file) in a manner that can be relied on,
except for
dismounting the volume.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

From: amitr0 [mailto:xxxxx@gmail.com] Posted At: Monday, June 21,
2010
10:19 AM Posted To: ntfsd
Conversation: Force Flushing FS MetaData
Subject: Force Flushing FS MetaData

Hello all,

is there a way to force flush the NTFS metadata to the disk at a
particular
time? I have tried ZwFlushBuffersFile but that doesnt seem to work
even though
it returns success.

Here is the scenario:

  1. i do a CreateFileW(pwcszFileName, GENERIC_READ | GENERIC_WRITE,
    FILE_SHARE_READ | FILE_SHARE_WRITE, 0, CREATE_ALWAYS,
    FILE_ATTRIBUTE_NORMAL |
    FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH, 0); 2. I set file
    size by
    SetFilePointerEx( hFile,liDistanceToMove,NULL, FILE_BEGIN); 3. set the
    EOF
    SetEndOfFile(hFile); 4. zwFlushBuffersFile(hFile,&IoStatusBlock);
  2. CloseHandle(hFile)

right after this, when I look at the MFT to see if the entry has been
created,
it doesn’t show it (we have debug code to parse NTFS correctly)

So is there a way via some IOCTL or otherwise to ask NTFS to flushthe
FS meta
data.

Would IOCTL_VOLSNAP_FLUSH_AND_HOLD_WRITES work?

thanks

amit

–

  • amitr0

__________ Information from ESET Smart Security, version of virus
signature
database 5215 (20100621) __________

The message was checked by ESET Smart Security.

http://www.eset.com

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars (including our
new fs
mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

–
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.

__________ Information from ESET Smart Security, version of virus
signature
database 5215 (20100621) __________

The message was checked by ESET Smart Security.

http://www.eset.com

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

6

Very interesting, Don.

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Monday, June 21, 2010 12:51 PM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] Force Flushing FS MetaData

Actually, those of us who track fault tolerance topics have found it is much
worse than that. First they are now putting operating systems on some
drives so they can have more intelligence, and guess what the OS’es crash
every so often. Second they are caching the data in high density DRAM
without ECC, and a Google published last year the failure rates on this is
somewhere between 1000 and 10000 times the data published by the memory chip
companies. Finally, the push for denser and denser drives, has upped the
error rates on accessing the drives is going up faster than what many
vendors are publishing.

Bottom line is the reliability of data stored on a hard drive (without raid
and a lot of backups) has decreased significantly in the last 10 years.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: Joseph M. Newcomer [mailto:xxxxx@flounder.com] Posted At:
Monday, June 21, 2010 12:40 PM Posted To: ntfsd
Conversation: Force Flushing FS MetaData
Subject: RE: Force Flushing FS MetaData

One disk drive vendor told me (and this is why you should never let
techies
onto the show floor): “When you flush data to the drive, it is flushed
to the
onboard cache of the drive, and the driver declares that the data has
been
successfully flushed. At some future time, this cached data is
written to the
platter. We have no way to let you guarantee that the data has been
flushed
to the platter.” So, I ask, “So if there is a power failure while the
data is
in the drive’s onboard cache, you can’t guarantee the integrity of the
data.
This would be particularly bad if I’m using a transacted database.
Right?”
Reply: “Yes, but then we just blame it on Microsoft”.
joe

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Monday, June 21, 2010 10:22 AM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] Force Flushing FS MetaData

As has been said many times in the past there is no way to force NTFS
to flush
any data (either meta or file) in a manner that can be relied on,
except for
dismounting the volume.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

From: amitr0 [mailto:xxxxx@gmail.com] Posted At: Monday, June 21,
2010
10:19 AM Posted To: ntfsd
Conversation: Force Flushing FS MetaData
Subject: Force Flushing FS MetaData

Hello all,

is there a way to force flush the NTFS metadata to the disk at a
particular
time? I have tried ZwFlushBuffersFile but that doesnt seem to work
even though
it returns success.

Here is the scenario:

  1. i do a CreateFileW(pwcszFileName, GENERIC_READ | GENERIC_WRITE,
    FILE_SHARE_READ | FILE_SHARE_WRITE, 0, CREATE_ALWAYS,
    FILE_ATTRIBUTE_NORMAL |
    FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH, 0); 2. I set file
    size by
    SetFilePointerEx( hFile,liDistanceToMove,NULL, FILE_BEGIN); 3. set the
    EOF
    SetEndOfFile(hFile); 4. zwFlushBuffersFile(hFile,&IoStatusBlock);
  2. CloseHandle(hFile)

right after this, when I look at the MFT to see if the entry has been
created,
it doesn’t show it (we have debug code to parse NTFS correctly)

So is there a way via some IOCTL or otherwise to ask NTFS to flushthe
FS meta
data.

Would IOCTL_VOLSNAP_FLUSH_AND_HOLD_WRITES work?

thanks

amit

–

  • amitr0

__________ Information from ESET Smart Security, version of virus
signature
database 5215 (20100621) __________

The message was checked by ESET Smart Security.

http://www.eset.com

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars (including our
new fs
mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

–
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.

__________ Information from ESET Smart Security, version of virus
signature
database 5215 (20100621) __________

The message was checked by ESET Smart Security.

http://www.eset.com

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars (including our new fs
mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

7

Sounds like Seagate …

Gary G. Little
H (952) 223-1349
C (952) 454-4629
xxxxx@comcast.net

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Joseph M. Newcomer
Sent: Monday, June 21, 2010 11:40 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Force Flushing FS MetaData

One disk drive vendor told me (and this is why you should never let techies
onto the show floor): “When you flush data to the drive, it is flushed to
the onboard cache of the drive, and the driver declares that the data has
been successfully flushed. At some future time, this cached data is written
to the platter. We have no way to let you guarantee that the data has been
flushed to the platter.” So, I ask, “So if there is a power failure while
the data is in the drive’s onboard cache, you can’t guarantee the integrity
of the data. This would be particularly bad if I’m using a transacted
database. Right?” Reply: “Yes, but then we just blame it on Microsoft”.
joe

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Monday, June 21, 2010 10:22 AM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] Force Flushing FS MetaData

As has been said many times in the past there is no way to force NTFS to
flush any data (either meta or file) in a manner that can be relied on,
except for dismounting the volume.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

From: amitr0 [mailto:xxxxx@gmail.com] Posted At: Monday, June 21, 2010
10:19 AM Posted To: ntfsd
Conversation: Force Flushing FS MetaData
Subject: Force Flushing FS MetaData

Hello all,
?
is there a way to force flush the NTFS metadata to the disk at a particular
time? I have tried ZwFlushBuffersFile but that doesnt seem to work even
though it returns success.
?
Here is the scenario:
?

  1. i do a CreateFileW(pwcszFileName, GENERIC_READ | GENERIC_WRITE,
    FILE_SHARE_READ | FILE_SHARE_WRITE, 0, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL
    | FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH, 0); 2. I set file size
    by SetFilePointerEx( hFile,liDistanceToMove,NULL, FILE_BEGIN); 3. set the
    EOF SetEndOfFile(hFile); 4. zwFlushBuffersFile(hFile,&IoStatusBlock);
  2. CloseHandle(hFile)
    ?
    right after this, when I look at the MFT to see if the entry has been
    created, it doesn’t show it (we have debug code to parse NTFS correctly)
    ?
    So is there a way via some IOCTL or otherwise to ask NTFS to flushthe FS
    meta data.
    ?
    ?
    Would IOCTL_VOLSNAP_FLUSH_AND_HOLD_WRITES work?
    ?
    thanks
    ?
    amit
    ?

–

  • amitr0

__________ Information from ESET Smart Security, version of virus signature
database 5215 (20100621) __________

The message was checked by ESET Smart Security.

http://www.eset.com

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars (including our new fs
mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

–
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

8

> Reply: “Yes, but then we just blame it on Microsoft”.

Sadly that’s an all too common response in many fields, but the
storage industry does seem to use it all too often.

My favourite was a well known HBA vendor commissioned to write an
MPIO driver who couldn’t work out how the SCSI ALUA commands
functioned. It was only a minor part of the overall project but as
time became short on the project their excuses became ever more
extravagant until they finally asserted (with a straight face) “It
will never work until Microsoft re-write PnP”.

That one caused an almighty stink :slight_smile:

Mark.

At 17:40 21/06/2010, Joseph M. Newcomer wrote:

One disk drive vendor told me (and this is why you should never let techies
onto the show floor): “When you flush data to the drive, it is flushed to
the onboard cache of the drive, and the driver declares that the data has
been successfully flushed. At some future time, this cached data is written
to the platter. We have no way to let you guarantee that the data has been
flushed to the platter.” So, I ask, “So if there is a power failure while
the data is in the drive’s onboard cache, you can’t guarantee the integrity
of the data. This would be particularly bad if I’m using a transacted
database. Right?” Reply: “Yes, but then we just blame it on Microsoft”.
joe

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Monday, June 21, 2010 10:22 AM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] Force Flushing FS MetaData

As has been said many times in the past there is no way to force NTFS to
flush any data (either meta or file) in a manner that can be relied on,
except for dismounting the volume.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

From: amitr0 [mailto:xxxxx@gmail.com]
Posted At: Monday, June 21, 2010 10:19 AM
Posted To: ntfsd
Conversation: Force Flushing FS MetaData
Subject: Force Flushing FS MetaData

Hello all,

is there a way to force flush the NTFS metadata to the disk at a
particular time? I have tried ZwFlushBuffersFile but that doesnt seem to
work even though it returns success.

Here is the scenario:

  1. i do a CreateFileW(pwcszFileName, GENERIC_READ | GENERIC_WRITE,
    FILE_SHARE_READ | FILE_SHARE_WRITE, 0, CREATE_ALWAYS,
    FILE_ATTRIBUTE_NORMAL | FILE_FLAG_NO_BUFFERING |
    FILE_FLAG_WRITE_THROUGH, 0);
  2. I set file size by SetFilePointerEx( hFile,liDistanceToMove,NULL,
    FILE_BEGIN);
  3. set the EOF SetEndOfFile(hFile);
  4. zwFlushBuffersFile(hFile,&IoStatusBlock);
  5. CloseHandle(hFile)

right after this, when I look at the MFT to see if the entry has been
created, it doesn’t show it (we have debug code to parse NTFS correctly)

So is there a way via some IOCTL or otherwise to ask NTFS to flushthe FS
meta data.

Would IOCTL_VOLSNAP_FLUSH_AND_HOLD_WRITES work?

thanks

amit

–

  • amitr0

__________ Information from ESET Smart Security, version of virus
signature database 5215 (20100621) __________

The message was checked by ESET Smart Security.

http://www.eset.com

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

–
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

9

I probably shouldn’t say which vendor it was, but think about how many
choices there are…not a lot…
joe

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Gary G. Little
Sent: Monday, June 21, 2010 4:53 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Force Flushing FS MetaData

Sounds like Seagate …=20

Gary G. Little
H (952) 223-1349
C (952) 454-4629
xxxxx@comcast.net

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Joseph M. =
Newcomer
Sent: Monday, June 21, 2010 11:40 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Force Flushing FS MetaData

One disk drive vendor told me (and this is why you should never let =
techies
onto the show floor): “When you flush data to the drive, it is flushed =
to
the onboard cache of the drive, and the driver declares that the data =
has
been successfully flushed. At some future time, this cached data is =
written
to the platter. We have no way to let you guarantee that the data has =
been
flushed to the platter.” So, I ask, “So if there is a power failure =
while
the data is in the drive’s onboard cache, you can’t guarantee the =
integrity
of the data. This would be particularly bad if I’m using a transacted
database. Right?” Reply: “Yes, but then we just blame it on Microsoft”.
joe

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Monday, June 21, 2010 10:22 AM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] Force Flushing FS MetaData

As has been said many times in the past there is no way to force NTFS to
flush any data (either meta or file) in a manner that can be relied on,
except for dismounting the volume.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

From: amitr0 [mailto:xxxxx@gmail.com] Posted At: Monday, June 21, =
2010
10:19 AM Posted To: ntfsd
Conversation: Force Flushing FS MetaData
Subject: Force Flushing FS MetaData

Hello all,
=A0
is there a way to force flush the NTFS metadata to the disk at a =
particular
time? I have tried ZwFlushBuffersFile but that doesnt seem to work even
though it returns success.
=A0
Here is the scenario:
=A0

  1. i do a CreateFileW(pwcszFileName, GENERIC_READ | GENERIC_WRITE,
    FILE_SHARE_READ | FILE_SHARE_WRITE, 0, CREATE_ALWAYS, =
    FILE_ATTRIBUTE_NORMAL
    | FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH, 0); 2. I set file =
    size
    by SetFilePointerEx( hFile,liDistanceToMove,NULL, FILE_BEGIN); 3. set =
    the
    EOF SetEndOfFile(hFile); 4. zwFlushBuffersFile(hFile,&IoStatusBlock);
  2. CloseHandle(hFile)
    =A0
    right after this, when I look at the MFT to see if the entry has been
    created, it doesn’t show it (we have debug code to parse NTFS correctly)
    =A0
    So is there a way via some IOCTL or otherwise to ask NTFS to flushthe FS
    meta data.
    =A0
    =A0
    Would IOCTL_VOLSNAP_FLUSH_AND_HOLD_WRITES work?
    =A0
    thanks
    =A0
    amit
    =A0

–

  • amitr0

__________ Information from ESET Smart Security, version of virus =
signature
database 5215 (20100621) __________

The message was checked by ESET Smart Security.

http://www.eset.com

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars (including our =
new fs
mini-filter seminar) visit:=20
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=3DListServer

–
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:=20
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=3DListServer

NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:=20
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=3DListServer

–=20
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

10

dismounting thee system volume is not a vey bright idea :smiley: (pun intended)

On Mon, Jun 21, 2010 at 10:36 PM, Mark Cariddi wrote:

> If you could dismount the volume, it would do a flush.
>
> --Mark Cariddi
> OSR, Open Systems Resources, Inc
>
> -----Original Message-----
> From: xxxxx@lists.osr.com [mailto:
> xxxxx@lists.osr.com] On Behalf Of Don Burn
> Sent: Monday, June 21, 2010 12:51 PM
> To: Windows File Systems Devs Interest List
> Subject: RE:[ntfsd] Force Flushing FS MetaData
>
> Actually, those of us who track fault tolerance topics have found it is
> much worse than that. First they are now putting operating systems on
> some drives so they can have more intelligence, and guess what the OS’es
> crash every so often. Second they are caching the data in high density
> DRAM without ECC, and a Google published last year the failure rates on
> this is somewhere between 1000 and 10000 times the data published by the
> memory chip companies. Finally, the push for denser and denser drives,
> has upped the error rates on accessing the drives is going up faster
> than what many vendors are publishing.
>
> Bottom line is the reliability of data stored on a hard drive (without
> raid and a lot of backups) has decreased significantly in the last 10
> years.
>
>
> Don Burn (MVP, Windows DKD)
> Windows Filesystem and Driver Consulting
> Website: http://www.windrvr.com
> Blog: http://msmvps.com/blogs/WinDrvr
>
>
>
>
> > -----Original Message-----
> > From: Joseph M. Newcomer [mailto:xxxxx@flounder.com]
> > Posted At: Monday, June 21, 2010 12:40 PM
> > Posted To: ntfsd
> > Conversation: Force Flushing FS MetaData
> > Subject: RE: Force Flushing FS MetaData
> >
> > One disk drive vendor told me (and this is why you should never let
> techies
> > onto the show floor): “When you flush data to the drive, it is flushed
> to the
> > onboard cache of the drive, and the driver declares that the data has
> been
> > successfully flushed. At some future time, this cached data is
> written to the
> > platter. We have no way to let you guarantee that the data has been
> flushed
> > to the platter.” So, I ask, “So if there is a power failure while the
> data is
> > in the drive’s onboard cache, you can’t guarantee the integrity of the
> data.
> > This would be particularly bad if I’m using a transacted database.
> Right?”
> > Reply: “Yes, but then we just blame it on Microsoft”.
> > joe
> >
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
> > Sent: Monday, June 21, 2010 10:22 AM
> > To: Windows File Systems Devs Interest List
> > Subject: RE:[ntfsd] Force Flushing FS MetaData
> >
> > As has been said many times in the past there is no way to force NTFS
> to flush
> > any data (either meta or file) in a manner that can be relied on,
> except for
> > dismounting the volume.
> >
> >
> > Don Burn (MVP, Windows DKD)
> > Windows Filesystem and Driver Consulting
> > Website: http://www.windrvr.com
> > Blog: http://msmvps.com/blogs/WinDrvr
> >
> >
> >
> >
> > From: amitr0 [mailto:xxxxx@gmail.com] Posted At: Monday, June 21,
> 2010
> > 10:19 AM Posted To: ntfsd
> > Conversation: Force Flushing FS MetaData
> > Subject: Force Flushing FS MetaData
> >
> > Hello all,
> >
> > is there a way to force flush the NTFS metadata to the disk at a
> particular
> > time? I have tried ZwFlushBuffersFile but that doesnt seem to work
> even though
> > it returns success.
> >
> > Here is the scenario:
> >
> > 1. i do a CreateFileW(pwcszFileName, GENERIC_READ | GENERIC_WRITE,
> > FILE_SHARE_READ | FILE_SHARE_WRITE, 0, CREATE_ALWAYS,
> FILE_ATTRIBUTE_NORMAL |
> > FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH, 0); 2. I set file
> size by
> > SetFilePointerEx( hFile,liDistanceToMove,NULL, FILE_BEGIN); 3. set the
> EOF
> > SetEndOfFile(hFile); 4. zwFlushBuffersFile(hFile,&IoStatusBlock);
> > 5. CloseHandle(hFile)
> >
> > right after this, when I look at the MFT to see if the entry has been
> created,
> > it doesn’t show it (we have debug code to parse NTFS correctly)
> >
> > So is there a way via some IOCTL or otherwise to ask NTFS to flushthe
> FS meta
> > data.
> >
> >
> > Would IOCTL_VOLSNAP_FLUSH_AND_HOLD_WRITES work?
> >
> > thanks
> >
> > amit
> >
> >
> > –
> >
> > - amitr0
> >
> > Information from ESET Smart Security, version of virus
> signature
> > database 5215 (20100621)
> >
> > The message was checked by ESET Smart Security.
> >
> > http://www.eset.com
> >
> >
> > —
> > NTFSD is sponsored by OSR
> >
> > For our schedule of debugging and file system seminars (including our
> new fs
> > mini-filter seminar) visit:
> > http://www.osr.com/seminars
> >
> > To unsubscribe, visit the List Server section of OSR Online at
> > http://www.osronline.com/page.cfm?name=ListServer
> >
> > –
> > This message has been scanned for viruses and dangerous content by
> > MailScanner, and is believed to be clean.
> >
> >
> >
> > Information from ESET Smart Security, version of virus
> signature
> > database 5215 (20100621)
> >
> > The message was checked by ESET Smart Security.
> >
> > http://www.eset.com
> >
>
>
> —
> NTFSD is sponsored by OSR
>
> For our schedule of debugging and file system seminars
> (including our new fs mini-filter seminar) visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> —
> NTFSD is sponsored by OSR
>
> For our schedule of debugging and file system seminars
> (including our new fs mini-filter seminar) visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

–

- amitr0