Query about CreateFile API with FILE_FLAG_NO_BUFFERING flag

NTFSD
1

Hi All,

While reading information about CreateFile API, I have came across the
flag FILE_FLAG_NO_BUFFERING which state that it prevents the file contents
from being cached, but the file metadata may still be cached.

And if you want that all the data to be written to a file with it’s metadata
you can use FlushFileBuffers API.

MSDN also states FILE_FLAG_NO_BUFFERING flag does not affect hard disk caching.

1> Does that mean any data written to the file may not get reflected to the disk
even if you have used FILE_FLAG_NO_BUFFERING in case of power failures?

2> Does the point 1 apply to FlushFileBuffers API? i.e. FlushFileBuffers is called and still it is cached by hard disk caching?

3> Can any one explain me what is the difference between using FILE_FLAG_NO_BUFFERING flag and using FlushFileBuffers API with respect to disk caching?

Thanks & Regards,
Amit.

2

Disk cache is buffer on disk (on the actual piece of hardware). Data are
stored there just for a short time. This buffer is always used for reading
and writing into disk and you cannot change this behaviour by setting
FILE_FLAG_NO_BUFFERING.

wrote in message news:xxxxx@ntfsd…
> Hi All,
>
> While reading information about CreateFile API, I have came across the
> flag FILE_FLAG_NO_BUFFERING which state that it prevents the file contents
> from being cached, but the file metadata may still be cached.
>
> And if you want that all the data to be written to a file with it’s
metadata
> you can use FlushFileBuffers API.
>
> MSDN also states FILE_FLAG_NO_BUFFERING flag does not affect hard disk
caching.
>
> 1> Does that mean any data written to the file may not get reflected to
the disk
> even if you have used FILE_FLAG_NO_BUFFERING in case of power failures?
>
> 2> Does the point 1 apply to FlushFileBuffers API? i.e. FlushFileBuffers
is called and still it is cached by hard disk caching?
>
> 3> Can any one explain me what is the difference between using
FILE_FLAG_NO_BUFFERING flag and using FlushFileBuffers API with respect to
disk caching?
>
> Thanks & Regards,
> Amit.
>

3

The only way to disable this is to disable write caching on the devnode for a disk, which is a global setting.

Another option is to use a quality RAID controller with a battery-backed write cache that will survive a power loss up to many hours.

? S

-----Original Message-----
From: Jan Milan
Sent: Wednesday, January 14, 2009 08:35
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Query about CreateFile API with FILE_FLAG_NO_BUFFERING flag

Disk cache is buffer on disk (on the actual piece of hardware). Data are
stored there just for a short time. This buffer is always used for reading
and writing into disk and you cannot change this behaviour by setting
FILE_FLAG_NO_BUFFERING.

wrote in message news:xxxxx@ntfsd…
> Hi All,
>
> While reading information about CreateFile API, I have came across the
> flag FILE_FLAG_NO_BUFFERING which state that it prevents the file contents
> from being cached, but the file metadata may still be cached.
>
> And if you want that all the data to be written to a file with it’s
metadata
> you can use FlushFileBuffers API.
>
> MSDN also states FILE_FLAG_NO_BUFFERING flag does not affect hard disk
caching.
>
> 1> Does that mean any data written to the file may not get reflected to
the disk
> even if you have used FILE_FLAG_NO_BUFFERING in case of power failures?
>
> 2> Does the point 1 apply to FlushFileBuffers API? i.e. FlushFileBuffers
is called and still it is cached by hard disk caching?
>
> 3> Can any one explain me what is the difference between using
FILE_FLAG_NO_BUFFERING flag and using FlushFileBuffers API with respect to
disk caching?
>
> Thanks & Regards,
> Amit.
>

—
NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@valhallalegends.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

4

>

1> Does that mean any data written to the file may not get reflected to the disk
even if you have used FILE_FLAG_NO_BUFFERING in case of power failures?

Perhaps. But after WriteFile (or ZwWriteFile) returns successfully the
written data has already been successfully sent to the storage stack
(and it returned STATUS_SUCCESS too).

2> Does the point 1 apply to FlushFileBuffers API? i.e. FlushFileBuffers is called and still it is cached by hard disk caching?

Yes, it does.

3> Can any one explain me what is the difference between using FILE_FLAG_NO_BUFFERING flag and using FlushFileBuffers API with respect to disk caching?

FlushFileBuffers just flushes not written yet data from the system
cache. The data are sent to the storage stack in the same way as
FILE_FLAG_NO_BUFFERING uses.
BTW using FILE_FLAG_NO_BUFFERING it’s possible only to read/write data
blocks that are multiple to sector size.

–
Best regards,
Vladimir Zinin
xxxxx@eldos.com

5

> BTW using FILE_FLAG_NO_BUFFERING it’s possible only to read/write data

blocks that are multiple to sector size.

…and all of the 3: offset, length and pointer value must be aligned to sector size.

–

Maxim S. Shatskih
Windows DDK MVP

xxxxx@storagecraft.com

http://www.storagecraft.com

6

Note that hard drives have onboard caches also. If the OS successfully
flushes data to the onboard cache, there is still no guarantee that it is on
the platters. Furthermore, many of the drives do not support operations to
flush their caches to the platters. When I asked one vendor about this,
their response was “we have no reason to do this; it would cause too many
problems.” So when I said “so how do I know my transacted file system is
correctly updated?” the answer was “If it fails, we just blame Microsoft”
[Lesson: don’t let the techies talk to customers on the trade show floor…]
This was a few years ago; perhaps they have learned their lesson and support
this now. But it makes me very nervous about my server…
joe

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Vladimir Zinin
Sent: Wednesday, January 14, 2009 2:15 PM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Query about CreateFile API with FILE_FLAG_NO_BUFFERING
flag

1> Does that mean any data written to the file may not get reflected
1> to the disk
even if you have used FILE_FLAG_NO_BUFFERING in case of power failures?

Perhaps. But after WriteFile (or ZwWriteFile) returns successfully the
written data has already been successfully sent to the storage stack (and it
returned STATUS_SUCCESS too).

2> Does the point 1 apply to FlushFileBuffers API? i.e. FlushFileBuffers
is called and still it is cached by hard disk caching?

Yes, it does.

3> Can any one explain me what is the difference between using
FILE_FLAG_NO_BUFFERING flag and using FlushFileBuffers API with respect to
disk caching?

FlushFileBuffers just flushes not written yet data from the system cache.
The data are sent to the storage stack in the same way as
FILE_FLAG_NO_BUFFERING uses.
BTW using FILE_FLAG_NO_BUFFERING it’s possible only to read/write data
blocks that are multiple to sector size.

–
Best regards,
Vladimir Zinin
xxxxx@eldos.com

NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@flounder.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

–
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

7

It is a given in filesystem development that the people who build disks are
out to get you :slight_smile:

One serious point to note is that filesystems cache metadata and data and
that it is not as unreasonable as it sounds to send MJ_FLUSH_BUFFERS to file
objects which are opened NO_BUFFERING (or WRITE_THROUGH) to try to ensure
that the data is on disk.

Rod
“Joseph M. Newcomer” wrote in message
news:xxxxx@ntfsd…
> Note that hard drives have onboard caches also. If the OS successfully
> flushes data to the onboard cache, there is still no guarantee that it is
> on
> the platters. Furthermore, many of the drives do not support operations
> to
> flush their caches to the platters. When I asked one vendor about this,
> their response was “we have no reason to do this; it would cause too many
> problems.” So when I said “so how do I know my transacted file system is
> correctly updated?” the answer was “If it fails, we just blame Microsoft”
> [Lesson: don’t let the techies talk to customers on the trade show
> floor…]
> This was a few years ago; perhaps they have learned their lesson and
> support
> this now. But it makes me very nervous about my server…
> joe
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Vladimir Zinin
> Sent: Wednesday, January 14, 2009 2:15 PM
> To: Windows File Systems Devs Interest List
> Subject: Re:[ntfsd] Query about CreateFile API with FILE_FLAG_NO_BUFFERING
> flag
>
>
>>
>> 1> Does that mean any data written to the file may not get reflected
>> 1> to the disk
>> even if you have used FILE_FLAG_NO_BUFFERING in case of power failures?
>>
>
> Perhaps. But after WriteFile (or ZwWriteFile) returns successfully the
> written data has already been successfully sent to the storage stack (and
> it
> returned STATUS_SUCCESS too).
>
>> 2> Does the point 1 apply to FlushFileBuffers API? i.e. FlushFileBuffers
> is called and still it is cached by hard disk caching?
>>
>
> Yes, it does.
>
>> 3> Can any one explain me what is the difference between using
> FILE_FLAG_NO_BUFFERING flag and using FlushFileBuffers API with respect to
> disk caching?
>>
>
> FlushFileBuffers just flushes not written yet data from the system cache.
> The data are sent to the storage stack in the same way as
> FILE_FLAG_NO_BUFFERING uses.
> BTW using FILE_FLAG_NO_BUFFERING it’s possible only to read/write data
> blocks that are multiple to sector size.
>
> –
> Best regards,
> Vladimir Zinin
> xxxxx@eldos.com
>
> —
> NTFSD is sponsored by OSR
>
> For our schedule debugging and file system seminars
> (including our new fs mini-filter seminar) visit:
> http://www.osr.com/seminars
>
> You are currently subscribed to ntfsd as: xxxxx@flounder.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
> –
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>

8

> Disk cache is buffer on disk (on the actual piece of hardware). Data

are stored there just for a short time.

This (stored there for a short time) is true only in single-disk desktop
systems. When we start talking about storage arrays (and you use the same
APIs to write to them), this statement is not only false, but also quite
misleading. Arrays may buffer data for long periods of time (hours) under
the right conditions. USUALLY they will write through the cache and the data
actually makes it to the rotating (or stable) storage fairly quickly, but
even this is not guaranteed. Most arrays will have cache that is battery
backed and will make some guarantees about data getting to the stable
storage before running out of power. If they don’t make those guarantees,
run screaming from them.

The point is that to think that the way an IDE or SATA disk on a PC handles
data is typical is wrong. There are lots more ways the hardware might be
swizzling the data underneath that innocent transfer to the disk…

…dave-+

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:bounce-350745-
xxxxx@lists.osr.com] On Behalf Of Jan Milan
Sent: Wednesday, January 14, 2009 9:35 AM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Query about CreateFile API with
FILE_FLAG_NO_BUFFERING flag

Disk cache is buffer on disk (on the actual piece of hardware). Data
are
stored there just for a short time. This buffer is always used for
reading
and writing into disk and you cannot change this behaviour by setting
FILE_FLAG_NO_BUFFERING.

wrote in message news:xxxxx@ntfsd…
> > Hi All,
> >
> > While reading information about CreateFile API, I have came across
> the
> > flag FILE_FLAG_NO_BUFFERING which state that it prevents the file
> contents
> > from being cached, but the file metadata may still be cached.
> >
> > And if you want that all the data to be written to a file with it’s
> metadata
> > you can use FlushFileBuffers API.
> >
> > MSDN also states FILE_FLAG_NO_BUFFERING flag does not affect hard
> disk
> caching.
> >
> > 1> Does that mean any data written to the file may not get reflected
> to
> the disk
> > even if you have used FILE_FLAG_NO_BUFFERING in case of power
> failures?
> >
> > 2> Does the point 1 apply to FlushFileBuffers API? i.e.
> FlushFileBuffers
> is called and still it is cached by hard disk caching?
> >
> > 3> Can any one explain me what is the difference between using
> FILE_FLAG_NO_BUFFERING flag and using FlushFileBuffers API with respect
> to
> disk caching?
> >
> > Thanks & Regards,
> > Amit.
> >
>
>
>
> —
> NTFSD is sponsored by OSR
>
> For our schedule debugging and file system seminars
> (including our new fs mini-filter seminar) visit:
> http://www.osr.com/seminars
>
> You are currently subscribed to ntfsd as: xxxxx@exmsft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com