Interaction of File Systems with Bulk Disk Access

Hello friends,

My question involves the discrepancies in time I see when copying data
from bulk storage (HDD, SSD, CD/DVD). If I do a transfer at a level
lower than the filesystem (e.g. read \.\CDROM0) I seem to get much
faster speeds than by copying the files themselves, to the tune of a
5x increase in speed.

It seems to me the creation and management of the files is what adds
the overhead. Can anyone verify this, or propose an alternative? I
have noticed something similar with FLOSS projects I use; their
directory structures cause lots of lag due to IO and the IO they
perform takes a long time. There are typically many files.

My apologies for not having good timing data to discuss. I will see
what I can come up with. If anyone can suggest a methodology I would
appreciate it, it is not immediately obvious to me how I would compare
same data transfer sizes.

I realize this question may not be entirely appropriate, but I hope to
get a good answer here.

Cheers,
R0b0t1

On Mon, Apr 2, 2018 at 1:00 PM, R0b0t1 wrote:
> Hello friends,
>
> If I do a transfer at a level
> lower than the filesystem (e.g. read \.\CDROM0) I seem to get much
> faster speeds than by copying the files themselves, to the tune of a
> 5x increase in speed.
>

Sorry, the increase is more like 10-20x. 20 minutes vs. 3+ hours.

Cheers,
R0b0t1

I find it highly unlikely. Only if your file access is highly inefficient because you’re using wrong access flags and patterns

On Mon, Apr 2, 2018 at 5:19 PM, xxxxx@broadcom.com
wrote:
> I find it highly unlikely. Only if your file access is highly inefficient because you’re using wrong access flags and patterns
>

Are the right ones documented anywhere? If anyone wants to reproduce,
populate a directory with lots of files with small names. I’ll need to
check if it is just very “wide” directories with this problem, or any
fixed amount under a directory tree.

> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:>

I?m not sure I understand your question.

Does it not reduce to:

If I copy a raw disk, directly, in big chunks… should this be faster than copying the same data from that disk first located and maintained via the file system and then read by the chance manager?

Is it not natural and obvious that the file system and rest of the system will add overhead as they add additional ?value??

This is the reason that many backup products do direct disk reads (first reading the storage bitmap, then reading those sectors in large chunks), I stated of doing file by file copy.

If you bypass the file system and VM subsystem, you naturally have less overhead this if you use them, right?

Peter
OSR
@OSRDrivers

xxxxx@osr.com wrote:

I?m not sure I understand your question.

Does it not reduce to:

If you bypass the file system and VM subsystem, you naturally have less overhead this if you use them, right?

More than that, his original question mentioned a CD-ROM device. 
CD-ROMs have absolutely dreadful random seek times.  I’m not at all to
surprised to find a 10x or 20x performance improvement by streaming
to/from a CD.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

On Tue, Apr 3, 2018 at 1:22 PM, xxxxx@probo.com wrote:
> xxxxx@osr.com wrote:
>> I?m not sure I understand your question.
>>
>> Does it not reduce to:
>> …
>> If you bypass the file system and VM subsystem, you naturally have less overhead this if you use them, right?
>
> More than that, his original question mentioned a CD-ROM device.
> CD-ROMs have absolutely dreadful random seek times. I’m not at all to
> surprised to find a 10x or 20x performance improvement by streaming
> to/from a CD.
>

Well, I think the explanations given are sufficient. I think my
confusion was from the lack of IOP shaping in the driver layers.

I have no idea what that even is.

Peter
OSR
@OSRDrivers

On Sun, Apr 8, 2018 at 2:59 PM, xxxxx@osr.com wrote:
>


>
> I have no idea what that even is.
>

IO requests are rescheduled to improve performance. Newer drive
control firmware also helps, by rescheduling writes so that e.g. a
read of A, C, B does not move the head A -> C -> B but instead moves
the head A -> B -> C (assuming A, B, and C are linear in space). My
understanding is that the OS has to hold back requests instead of
firing them off immediately for this to work well.

It can also be used in a way similar to packet shaping, but for QoS
management of disk bandwidth.

Cheers,
R0b0t1

We call that ?seek optimization? ? I?ve written a whole lot of that code in my career… but none for hard drives in the last decade plus. In my experience, this type of optimization is seriously out of date, because we can no longer guess at the true geometry of the drive. So, ?nearest sector first? and ?elevator service / next sector first? based on Starting LBA are no longer very effective. Windows has never had any such code in the OS (though it is entirely possible that one or two of the original disk drivers ? maybe the floppy driver ? MAY have had such code in them at some point. I don?t remember). But I can authoritatively tell you that no such code exists in Windows today.

Peter
OSR
@OSRDrivers

Yep, modern drives do all this internally. Hence command queuing.

* Bob

? Bob Ammerman
? xxxxx@ramsystems.biz
? 716.864.8337

138 Liston St
Buffalo, NY 14223
www.ramsystems.biz

-----Original Message-----
From: xxxxx@lists.osr.com On Behalf Of xxxxx@osr.com
Sent: Sunday, April 8, 2018 7:28 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Interaction of File Systems with Bulk Disk Access



We call that ?seek optimization? ? I?ve written a whole lot of that code in my career… but none for hard drives in the last decade plus. In my experience, this type of optimization is seriously out of date, because we can no longer guess at the true geometry of the drive. So, ?nearest sector first? and ?elevator service / next sector first? based on Starting LBA are no longer very effective. Windows has never had any such code in the OS (though it is entirely possible that one or two of the original disk drivers ? maybe the floppy driver ? MAY have had such code in them at some point. I don?t remember). But I can authoritatively tell you that no such code exists in Windows today.

Peter
OSR
@OSRDrivers


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:>

>modern drives

Thanks for that addition, Mr. Ammerman. I had meant to mention that as well.

True for hard drives, and particularly true for SSDs… which have ?a whole lot of stuff going on? :slight_smile:

Peter
OSR
@OSRDrivers

Where “modern” is like at least the last 20 years or so.

Mark Roddy

On Sun, Apr 8, 2018 at 7:31 PM, xxxxx@ramsystems.biz
wrote:

> Yep, modern drives do all this internally. Hence command queuing.
>
> * Bob
>
>
> Bob Ammerman
> xxxxx@ramsystems.biz
> 716.864.8337
>
> 138 Liston St
> Buffalo, NY 14223
> www.ramsystems.biz
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com > osr.com> On Behalf Of xxxxx@osr.com
> Sent: Sunday, April 8, 2018 7:28 PM
> To: Windows System Software Devs Interest List
> Subject: RE:[ntdev] Interaction of File Systems with Bulk Disk Access
>
>


>
> We call that ?seek optimization? ? I?ve written a whole lot of that code
> in my career… but none for hard drives in the last decade plus. In my
> experience, this type of optimization is seriously out of date, because we
> can no longer guess at the true geometry of the drive. So, ?nearest sector
> first? and ?elevator service / next sector first? based on Starting LBA are
> no longer very effective. Windows has never had any such code in the OS
> (though it is entirely possible that one or two of the original disk
> drivers ? maybe the floppy driver ? MAY have had such code in them at some
> point. I don?t remember). But I can authoritatively tell you that no such
> code exists in Windows today.
>
> Peter
> OSR
> @OSRDrivers
>
>
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:> showlists.cfm?list=ntdev>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:> showlists.cfm?list=ntdev>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:></http:></http:></http:>

20 years or so for drives, some disk controllers were doing this over 35 years ago.

Don Burn

Windows Driver Consulting

Website: http: http://www.windrvr.com

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@gmail.com
Sent: Monday, April 09, 2018 9:31 AM
To: Windows System Software Devs Interest List
Subject: Re: RE:[ntdev] Interaction of File Systems with Bulk Disk Access

Where “modern” is like at least the last 20 years or so.

Mark Roddy

On Sun, Apr 8, 2018 at 7:31 PM, xxxxx@ramsystems.biz mailto:xxxxx > wrote:

Yep, modern drives do all this internally. Hence command queuing.

* Bob

Bob Ammerman
xxxxx@ramsystems.biz mailto:xxxxx
716.864.8337

138 Liston St
Buffalo, NY 14223
www.ramsystems.biz http:

-----Original Message-----
From: xxxxx@lists.osr.com mailto:xxxxx > On Behalf Of xxxxx@osr.com mailto:xxxxx
Sent: Sunday, April 8, 2018 7:28 PM
To: Windows System Software Devs Interest List >
Subject: RE:[ntdev] Interaction of File Systems with Bulk Disk Access



We call that ?seek optimization? ? I?ve written a whole lot of that code in my career… but none for hard drives in the last decade plus. In my experience, this type of optimization is seriously out of date, because we can no longer guess at the true geometry of the drive. So, ?nearest sector first? and ?elevator service / next sector first? based on Starting LBA are no longer very effective. Windows has never had any such code in the OS (though it is entirely possible that one or two of the original disk drivers ? maybe the floppy driver ? MAY have had such code in them at some point. I don?t remember). But I can authoritatively tell you that no such code exists in Windows today.

Peter
OSR
@OSRDrivers


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:

— NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers! Details at To unsubscribe, visit the List Server section of OSR Online at</http:></http:></http:></http:></http:></http:></mailto:xxxxx></mailto:xxxxx></http:></mailto:xxxxx></mailto:xxxxx></http:>

There is quite a bit of interest in kernelspace reordering on Linux, which
is what I was trying to refer to. For various reasons the firmware based
optimizations are insufficient. So, if Windows does no optimizations, I
think that answers my question.

On Mon, Apr 9, 2018 at 8:40 AM, xxxxx@windrvr.com
wrote:
> 20 years or so for drives, some disk controllers were doing this over 35
> years ago.
>
>
>
>
>
> Don Burn
>
> Windows Driver Consulting
>
> Website: http://www.windrvr.com
>
>
>
>
>
>
>
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of
> xxxxx@gmail.com
> Sent: Monday, April 09, 2018 9:31 AM
> To: Windows System Software Devs Interest List
> Subject: Re: RE:[ntdev] Interaction of File Systems with Bulk Disk Access
>
>
>
> Where “modern” is like at least the last 20 years or so.
>
>
> Mark Roddy
>
>
>
> On Sun, Apr 8, 2018 at 7:31 PM, xxxxx@ramsystems.biz
> wrote:
>
> Yep, modern drives do all this internally. Hence command queuing.
>
> * Bob
>
>
> Bob Ammerman
> xxxxx@ramsystems.biz
> 716.864.8337
>
> 138 Liston St
> Buffalo, NY 14223
> www.ramsystems.biz
>
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> On Behalf Of xxxxx@osr.com
> Sent: Sunday, April 8, 2018 7:28 PM
> To: Windows System Software Devs Interest List
> Subject: RE:[ntdev] Interaction of File Systems with Bulk Disk Access
>
>


>
> We call that ?seek optimization? ? I?ve written a whole lot of that code
in
> my career… but none for hard drives in the last decade plus. In my
> experience, this type of optimization is seriously out of date, because we
> can no longer guess at the true geometry of the drive. So, ?nearest
sector
> first? and ?elevator service / next sector first? based on Starting LBA
are
> no longer very effective. Windows has never had any such code in the OS
> (though it is entirely possible that one or two of the original disk
drivers
> ? maybe the floppy driver ? MAY have had such code in them at some point.
I
> don?t remember). But I can authoritatively tell you that no such code
> exists in Windows today.
>
> Peter
> OSR
> @OSRDrivers
>
>
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at:
> http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
software
> drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at
> http:
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at:
> http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
software
> drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at
> http:
>
>
>
> — NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars
on
> crash dump analysis, WDF, Windows internals and software drivers! Details
at
> To unsubscribe, visit the List Server section of OSR Online at
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at:
> http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
software
> drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at
> http:</http:></http:></http:></http:></http:></http:></http:></http:></http:>

Actually if I recall correctly the adapter port drivers in windows had or
have some sort of scheduling mechanism, not sure if they still do. It
remains lame. The agents that understand how a specific disk hardware
operates are the agents closest to the disk hardware, generally the
firmware on the disk itself or in cases where the disks are tightly coupled
to the adapter(s), the adapter firmware.

Mark Roddy

On Mon, Apr 9, 2018 at 10:11 AM, xxxxx@gmail.com
wrote:

> There is quite a bit of interest in kernelspace reordering on Linux, which
> is what I was trying to refer to. For various reasons the firmware based
> optimizations are insufficient. So, if Windows does no optimizations, I
> think that answers my question.
>
>
> On Mon, Apr 9, 2018 at 8:40 AM, xxxxx@windrvr.com
> wrote:
> > 20 years or so for drives, some disk controllers were doing this over 35
> > years ago.
> >
> >
> >
> >
> >
> > Don Burn
> >
> > Windows Driver Consulting
> >
> > Website: http://www.windrvr.com
> >
> >
> >
> >
> >
> >
> >
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of
> > xxxxx@gmail.com
> > Sent: Monday, April 09, 2018 9:31 AM
> > To: Windows System Software Devs Interest List
> > Subject: Re: RE:[ntdev] Interaction of File Systems with Bulk Disk Access
> >
> >
> >
> > Where “modern” is like at least the last 20 years or so.
> >
> >
> > Mark Roddy
> >
> >
> >
> > On Sun, Apr 8, 2018 at 7:31 PM, xxxxx@ramsystems.biz
> > wrote:
> >
> > Yep, modern drives do all this internally. Hence command queuing.
> >
> > * Bob
> >
> >
> > Bob Ammerman
> > xxxxx@ramsystems.biz
> > 716.864.8337
> >
> > 138 Liston St
> https:
> > Buffalo, NY 14223
> https:
> > www.ramsystems.biz
> >
> >
> >
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > On Behalf Of xxxxx@osr.com
> > Sent: Sunday, April 8, 2018 7:28 PM
> > To: Windows System Software Devs Interest List
> > Subject: RE:[ntdev] Interaction of File Systems with Bulk Disk Access
> >
> >


> >
> > We call that ?seek optimization? ? I?ve written a whole lot of that code
> in
> > my career… but none for hard drives in the last decade plus. In my
> > experience, this type of optimization is seriously out of date, because
> we
> > can no longer guess at the true geometry of the drive. So, ?nearest
> sector
> > first? and ?elevator service / next sector first? based on Starting LBA
> are
> > no longer very effective. Windows has never had any such code in the OS
> > (though it is entirely possible that one or two of the original disk
> drivers
> > ? maybe the floppy driver ? MAY have had such code in them at some
> point. I
> > don?t remember). But I can authoritatively tell you that no such code
> > exists in Windows today.
> >
> > Peter
> > OSR
> > @OSRDrivers
> >
> >
> >
> >
> > —
> > NTDEV is sponsored by OSR
> >
> > Visit the list online at:
> > http:
> >
> > MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software
> > drivers!
> > Details at http:
> >
> > To unsubscribe, visit the List Server section of OSR Online at
> > http:
> >
> > —
> > NTDEV is sponsored by OSR
> >
> > Visit the list online at:
> > http:
> >
> > MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software
> > drivers!
> > Details at http:
> >
> > To unsubscribe, visit the List Server section of OSR Online at
> > http:
> >
> >
> >
> > — NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars
> on
> > crash dump analysis, WDF, Windows internals and software drivers!
> Details at
> > To unsubscribe, visit the List Server section of OSR Online at
> >
> >
> > —
> > NTDEV is sponsored by OSR
> >
> > Visit the list online at:
> > http:
> >
> > MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software
> > drivers!
> > Details at http:
> >
> > To unsubscribe, visit the List Server section of OSR Online at
> > http:
>
> — NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars
> on crash dump analysis, WDF, Windows internals and software drivers!
> Details at To unsubscribe, visit the List Server section of OSR Online at
></http:></http:></http:></http:></http:></http:></http:></http:></http:></https:></https:>

Some databases (Postgres, MySQL) have gotten good performance
increases by treating their backends more as storage controllers than
as simple file access. This leads to some strange performance
degradation on filesystems like ZFS, which also work similarly to a
database, but can not optimize access based on data content.

So it is not always knowledge of the physical media that counts.

On Mon, Apr 9, 2018 at 9:36 AM, xxxxx@gmail.com
wrote:
> Actually if I recall correctly the adapter port drivers in windows had or
> have some sort of scheduling mechanism, not sure if they still do. It
> remains lame. The agents that understand how a specific disk hardware
> operates are the agents closest to the disk hardware, generally the firmware
> on the disk itself or in cases where the disks are tightly coupled to the
> adapter(s), the adapter firmware.
>
> Mark Roddy
>
> On Mon, Apr 9, 2018 at 10:11 AM, xxxxx@gmail.com
> wrote:
>>
>> There is quite a bit of interest in kernelspace reordering on Linux, which
>> is what I was trying to refer to. For various reasons the firmware based
>> optimizations are insufficient. So, if Windows does no optimizations, I
>> think that answers my question.
>>
>>
>> On Mon, Apr 9, 2018 at 8:40 AM, xxxxx@windrvr.com
>> wrote:
>> > 20 years or so for drives, some disk controllers were doing this over 35
>> > years ago.
>> >
>> >
>> >
>> >
>> >
>> > Don Burn
>> >
>> > Windows Driver Consulting
>> >
>> > Website: http://www.windrvr.com
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > From: xxxxx@lists.osr.com
>> > [mailto:xxxxx@lists.osr.com] On Behalf Of
>> > xxxxx@gmail.com
>> > Sent: Monday, April 09, 2018 9:31 AM
>> > To: Windows System Software Devs Interest List
>> > Subject: Re: RE:[ntdev] Interaction of File Systems with Bulk Disk
>> > Access
>> >
>> >
>> >
>> > Where “modern” is like at least the last 20 years or so.
>> >
>> >
>> > Mark Roddy
>> >
>> >
>> >
>> > On Sun, Apr 8, 2018 at 7:31 PM, xxxxx@ramsystems.biz
>> > wrote:
>> >
>> > Yep, modern drives do all this internally. Hence command queuing.
>> >
>> > * Bob
>> >
>> >
>> > Bob Ammerman
>> > xxxxx@ramsystems.biz
>> > 716.864.8337
>> >
>> > 138 Liston St
>> > Buffalo, NY 14223
>> > www.ramsystems.biz
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: xxxxx@lists.osr.com
>> > On Behalf Of xxxxx@osr.com
>> > Sent: Sunday, April 8, 2018 7:28 PM
>> > To: Windows System Software Devs Interest List
>> > Subject: RE:[ntdev] Interaction of File Systems with Bulk Disk Access
>> >
>> >


>> >
>> > We call that ?seek optimization? ? I?ve written a whole lot of that code
>> > in
>> > my career… but none for hard drives in the last decade plus. In my
>> > experience, this type of optimization is seriously out of date, because
>> > we
>> > can no longer guess at the true geometry of the drive. So, ?nearest
>> > sector
>> > first? and ?elevator service / next sector first? based on Starting LBA
>> > are
>> > no longer very effective. Windows has never had any such code in the OS
>> > (though it is entirely possible that one or two of the original disk
>> > drivers
>> > ? maybe the floppy driver ? MAY have had such code in them at some
>> > point. I
>> > don?t remember). But I can authoritatively tell you that no such code
>> > exists in Windows today.
>> >
>> > Peter
>> > OSR
>> > @OSRDrivers
>> >
>> >
>> >
>> >
>> > —
>> > NTDEV is sponsored by OSR
>> >
>> > Visit the list online at:
>> > http:
>> >
>> > MONTHLY seminars on crash dump analysis, WDF, Windows internals and
>> > software
>> > drivers!
>> > Details at http:
>> >
>> > To unsubscribe, visit the List Server section of OSR Online at
>> > http:
>> >
>> > —
>> > NTDEV is sponsored by OSR
>> >
>> > Visit the list online at:
>> > http:
>> >
>> > MONTHLY seminars on crash dump analysis, WDF, Windows internals and
>> > software
>> > drivers!
>> > Details at http:
>> >
>> > To unsubscribe, visit the List Server section of OSR Online at
>> > http:
>> >
>> >
>> >
>> > — NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars
>> > on
>> > crash dump analysis, WDF, Windows internals and software drivers!
>> > Details at
>> > To unsubscribe, visit the List Server section of OSR Online at
>> >
>> >
>> > —
>> > NTDEV is sponsored by OSR
>> >
>> > Visit the list online at:
>> > http:
>> >
>> > MONTHLY seminars on crash dump analysis, WDF, Windows internals and
>> > software
>> > drivers!
>> > Details at http:
>> >
>> > To unsubscribe, visit the List Server section of OSR Online at
>> > http:
>>
>> — NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars
>> on crash dump analysis, WDF, Windows internals and software drivers! Details
>> at To unsubscribe, visit the List Server section of OSR Online at
>
>
> — NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars on
> crash dump analysis, WDF, Windows internals and software drivers! Details at
> To unsubscribe, visit the List Server section of OSR Online at</http:></http:></http:></http:></http:></http:></http:></http:></http:>

Hmmm… Most big and sophisticated database systems want the entire OS out of their way as much as possible. Basically, they REALLY want to be their own operating system. When it comes to disk storage, they definitely have no use for the file system provided by the OS. They can *always* do better… because they have something purpose-built for their needs.

Well, yes and no. Knowledge of the physical media ALWAYS counts… and the physical media is almost always being optimized by the internal drive logic. In general, the more reads/writes you give the drive in parallel, the better the optimization will be.

But… you’re right… knowledge of the physical media is not the ONLY thing that counts. Having an on disk structure that’s optimized for a given use can provide *tremendous* performance gains over a general purpose file system.

For example, years ago there was a system that needed to acquire satellite telemetry that was streaming non-stop at a fairly high rate. The data needed to be recorded without loss, and over a significant period of time.

The simple solution to this problem was to do raw (direct) I/O to the drives… and not even attempt to go through NTFS. I mean, if you’re going to dump an “enormous” chunk of data onto disk, and you’re always going to be the app that reads it back, you don’t really need a file system to keep track of your data blocks, your VDL, and when the last time was that you modified the data. You just wanna barf the data onto the disk, perhaps add a timestamp or two and some judicious checksumming, and you’re good to go. Bingo! You’ve created your own on disk structure. No file system required.

Peter
OSR
@OSRDrivers

Windows had IO request reordering code based on LBA back in the 2000 era, and it was beneficial to performance back then. Of course that was back when dinosaurs roamed the earth, so …

It was in scsiport, so not immediately obvious to the outside world. It was in the per-LUN queue and was c-scan like, but IIRC performed when pulling the next request from the queue (rather than when submitting). It’s been phased out in scsiport’s successors.

-p

-----Original Message-----
From: xxxxx@lists.osr.com On Behalf Of xxxxx@osr.com
Sent: Sunday, April 8, 2018 4:28 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Interaction of File Systems with Bulk Disk Access



We call that ?seek optimization? ? I?ve written a whole lot of that code in my career… but none for hard drives in the last decade plus. In my experience, this type of optimization is seriously out of date, because we can no longer guess at the true geometry of the drive. So, ?nearest sector first? and ?elevator service / next sector first? based on Starting LBA are no longer very effective. Windows has never had any such code in the OS (though it is entirely possible that one or two of the original disk drivers ? maybe the floppy driver ? MAY have had such code in them at some point. I don?t remember). But I can authoritatively tell you that no such code exists in Windows today.

Peter
OSR
@OSRDrivers


NTDEV is sponsored by OSR

Visit the list online at: https:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at https:

To unsubscribe, visit the List Server section of OSR Online at https:</https:></https:></https:>

>Windows had IO request reordering code based on LBA back in the 2000 era

Thanks, Peter. I appreciate the authoritative answer on this.

Peter
OSR
@OSRDrivers