The redirector doesn't cache SPARSE files

OSR_Community_User · February 4, 2002, 8:59pm

We have noticed that on W2K/SP2 the redirector does not attempt to cache if a file is marked with FILE_ATTRIBUTE_SPARSE_FILE, even though an oplock (batch or level 2) is granted. The file does not have to actually be sparse, we mark a regular file with the SPARSE attribute and the redirector doesn’t cache.

Here is what we see:

The routine MRxSmbSetSrvOpenFlags () is invoked after create is complete, to set the buffering/caching flags on a file, based on the oplock level which was granted. It’s defined as:
VOID
MRxSmbSetSrvOpenFlags (
PRX_CONTEXT RxContext,
RX_FILE_TYPE StorageType,
PMRX_SRV_OPEN SrvOpen,
PMRX_SMB_SRV_OPEN smbSrvOpen
)

The code that has been released with the IFSKIT doesn’t show any dependency on file attributes,
the disassembly, however, shows the following:

mrxsmb!MRxSmbSetSrvOpenFlags:
f31b3730 8b44240c mov eax,[esp+0xc]
f31b3734 8b4808 mov ecx,[eax+0x8]
f31b3737 83603800 and dword ptr [eax+0x38],0x0
f31b373b f6415902 test byte ptr [ecx+0x59],0x2 ds:0023:e23f0c01=00
f31b373f 750d jnz mrxsmb!MRxSmbSetSrvOpenFlags+0x44 (f31b374e)

eax points to MRX_SRV_OPEN, and ecx to MRX_FCB. The test statement tests the Attributes field for FILE_ATTRIBUTE_SPARSE_FILE, and does not set any of the caching/buffering flags if the file is marked as sparse.

We would appreciate your comments on this issue.

Sara

You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

OSR_Community_User · February 6, 2002, 9:17pm

Sara,

You are correct, the redirector currently does not cache files with the SPARSE attribute set. This was originally done because there was a problem with CC caching sparse files. We believe this issue has been resolved but we need to investigate and see if the redirector can be changed. Thanks for bringing this issue up.

As a side note, unlike the fastfat and cdfs samples, the redirector sample in the IFSKit does not match what is in the shipping product. It contains a subset of the actual functionality.

Neal Christiansen

This posting is provided “AS IS” with no warranties, and confers no rights."

-----Original Message-----
From: Sara Abraham [mailto:xxxxx@veritas.com]
Sent: Monday, February 04, 2002 06:08 PM
To: File Systems Developers
Subject: [ntfsd] The redirector doesn’t cache SPARSE files

We have noticed that on W2K/SP2 the redirector does not attempt to cache if a file is marked with FILE_ATTRIBUTE_SPARSE_FILE, even though an oplock (batch or level 2) is granted. The file does not have to actually be sparse, we mark a regular file with the SPARSE attribute and the redirector doesn’t cache.
?
Here is what we see:
?
The routine MRxSmbSetSrvOpenFlags () is invoked after create is complete, to set the buffering/caching flags on a file, based on the oplock level which was granted. It’s defined as:
VOID
MRxSmbSetSrvOpenFlags (
??? PRX_CONTEXT??? RxContext,
??? RX_FILE_TYPE??? StorageType,
??? PMRX_SRV_OPEN??? SrvOpen,
??? PMRX_SMB_SRV_OPEN?? smbSrvOpen
??? )

?The code that has been released with the IFSKIT doesn’t show any dependency on? file attributes,
the disassembly, however, shows the following:
?
mrxsmb!MRxSmbSetSrvOpenFlags:
f31b3730 8b44240c??? mov??? eax,[esp+0xc]
f31b3734 8b4808??? mov??? ecx,[eax+0x8]
f31b3737 83603800??? and??? dword ptr [eax+0x38],0x0
f31b373b f6415902??? test??? byte ptr [ecx+0x59],0x2 ds:0023:e23f0c01=00
f31b373f 750d??? jnz??? mrxsmb!MRxSmbSetSrvOpenFlags+0x44 (f31b374e)
?

eax points to MRX_SRV_OPEN, and ecx to MRX_FCB. The test statement tests the Attributes field for FILE_ATTRIBUTE_SPARSE_FILE, and does not set any of the caching/buffering flags if the file is marked as sparse.
?
We would appreciate your comments on this issue.
?
Sara

You are currently subscribed to ntfsd as: xxxxx@Windows.Microsoft.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

OSR_Community_User · February 6, 2002, 9:52pm

On another side note, it would be very helpful if the full LanMan
redirector source could be provided in the IFS kit at some point in
time. Sometimes it feels like I have spent more time stepping through
rdbss assembly than I have spent in my own filter; the redirector
behaves differently from fastfat/NTFS often enough that I think the
source would be a great debugging help.

Nicholas Ryan (xxxxx@nryan.com)

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:bounce-ntfsd-
xxxxx@lists.osr.com] On Behalf Of Neal Christiansen
Sent: Wednesday, February 06, 2002 6:12 PM
To: File Systems Developers
Subject: [ntfsd] RE: The redirector doesn’t cache SPARSE files

Sara,

You are correct, the redirector currently does not cache files with
the
SPARSE attribute set. This was originally done because there was a
problem with CC caching sparse files. We believe this issue has been
resolved but we need to investigate and see if the redirector can be
changed. Thanks for bringing this issue up.

As a side note, unlike the fastfat and cdfs samples, the redirector
sample
in the IFSKit does not match what is in the shipping product. It
contains
a subset of the actual functionality.

Neal Christiansen

This posting is provided “AS IS” with no warranties, and confers no
rights."

-----Original Message-----
From: Sara Abraham [mailto:xxxxx@veritas.com]
Sent: Monday, February 04, 2002 06:08 PM
To: File Systems Developers
Subject: [ntfsd] The redirector doesn’t cache SPARSE files

We have noticed that on W2K/SP2 the redirector does not attempt to
cache
if a file is marked with FILE_ATTRIBUTE_SPARSE_FILE, even though an
oplock
(batch or level 2) is granted. The file does not have to actually be
sparse, we mark a regular file with the SPARSE attribute and the
redirector doesn’t cache.

Here is what we see:

The routine MRxSmbSetSrvOpenFlags () is invoked after create is
complete,
to set the buffering/caching flags on a file, based on the oplock
level
which was granted. It’s defined as:
VOID
MRxSmbSetSrvOpenFlags (
??? PRX_CONTEXT??? RxContext,
??? RX_FILE_TYPE??? StorageType,
??? PMRX_SRV_OPEN??? SrvOpen,
??? PMRX_SMB_SRV_OPEN?? smbSrvOpen
??? )

?The code that has been released with the IFSKIT doesn’t show any
dependency on? file attributes,
the disassembly, however, shows the following:

mrxsmb!MRxSmbSetSrvOpenFlags:
f31b3730 8b44240c??? mov??? eax,[esp+0xc]
f31b3734 8b4808??? mov??? ecx,[eax+0x8]
f31b3737 83603800??? and??? dword ptr [eax+0x38],0x0
f31b373b f6415902??? test??? byte ptr [ecx+0x59],0x2
ds:0023:e23f0c01=00
f31b373f 750d??? jnz??? mrxsmb!MRxSmbSetSrvOpenFlags+0x44
(f31b374e)

eax points to MRX_SRV_OPEN, and ecx to MRX_FCB. The test statement
tests
the Attributes field for FILE_ATTRIBUTE_SPARSE_FILE, and does not set
any
of the caching/buffering flags if the file is marked as sparse.

We would appreciate your comments on this issue.

Sara

You are currently subscribed to ntfsd as: xxxxx@Windows.Microsoft.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntfsd as: xxxxx@secretseal.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

OSR_Community_User · February 7, 2002, 8:35pm

Sara,

This issue is not fixed in XP or .NET.

I was wrong when I earlier reported that there was an issue with CC and
sparse files. These types of files are not cached because of problems
handling out of disk space issues between the local and remote system.

Neal Christiansen

This posting is provided “AS IS” with no warranties, and confers no
rights."

-----Original Message-----
From: Sara Abraham [mailto:xxxxx@veritas.com]
Sent: Wednesday, February 06, 2002 08:52 PM
To: File Systems Developers
Subject: [ntfsd] RE: The redirector doesn’t cache SPARSE files

Thanks Neal,

We verified that this problem is fixed in XP and .NET. As for W2K, do
you
think that you’ll be able to provide a fix in SP3, and if so, what are
the
release dates ?
You say that there was a problem with CC caching SPARSE files. Was it
necessary to disable both read and write caching ? If disabling write
caching is sufficient, than a simple fix that allows read caching for
SPARSE
files would be good enough for us for now.

Sara

----- Original Message -----
From: “Neal Christiansen”
To: “File Systems Developers”
Sent: Wednesday, February 06, 2002 6:12 PM
Subject: [ntfsd] RE: The redirector doesn’t cache SPARSE files

Sara,

You are correct, the redirector currently does not cache files with the
SPARSE attribute set. This was originally done because there was a
problem
with CC caching sparse files. We believe this issue has been resolved
but
we need to investigate and see if the redirector can be changed. Thanks
for
bringing this issue up.

As a side note, unlike the fastfat and cdfs samples, the redirector
sample
in the IFSKit does not match what is in the shipping product. It
contains a
subset of the actual functionality.

Neal Christiansen

This posting is provided “AS IS” with no warranties, and confers no
rights."

-----Original Message-----
From: Sara Abraham [mailto:xxxxx@veritas.com]
Sent: Monday, February 04, 2002 06:08 PM
To: File Systems Developers
Subject: [ntfsd] The redirector doesn’t cache SPARSE files

We have noticed that on W2K/SP2 the redirector does not attempt to cache
if
a file is marked with FILE_ATTRIBUTE_SPARSE_FILE, even though an oplock
(batch or level 2) is granted. The file does not have to actually be
sparse,
we mark a regular file with the SPARSE attribute and the redirector
doesn’t
cache.

Here is what we see:

The routine MRxSmbSetSrvOpenFlags () is invoked after create is
complete, to
set the buffering/caching flags on a file, based on the oplock level
which
was granted. It’s defined as:
VOID
MRxSmbSetSrvOpenFlags (
PRX_CONTEXT RxContext,
RX_FILE_TYPE StorageType,
PMRX_SRV_OPEN SrvOpen,
PMRX_SMB_SRV_OPEN smbSrvOpen
)

The code that has been released with the IFSKIT doesn’t show any
dependency
on file attributes,
the disassembly, however, shows the following:

mrxsmb!MRxSmbSetSrvOpenFlags:
f31b3730 8b44240c mov eax,[esp+0xc]
f31b3734 8b4808 mov ecx,[eax+0x8]
f31b3737 83603800 and dword ptr [eax+0x38],0x0
f31b373b f6415902 test byte ptr [ecx+0x59],0x2 ds:0023:e23f0c01=00
f31b373f 750d jnz mrxsmb!MRxSmbSetSrvOpenFlags+0x44 (f31b374e)

eax points to MRX_SRV_OPEN, and ecx to MRX_FCB. The test statement tests
the
Attributes field for FILE_ATTRIBUTE_SPARSE_FILE, and does not set any of
the
caching/buffering flags if the file is marked as sparse.

We would appreciate your comments on this issue.

Sara
—
You are currently subscribed to ntfsd as: xxxxx@Windows.Microsoft.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: xxxxx@veritas.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: xxxxx@Windows.Microsoft.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

OSR_Community_User · February 8, 2002, 1:27am

Hi Neal,

It’s true that when we check the disassembly of MRxSmbSetSrvOpenFlags() on
XP, it shows that it’s still checking the SPARSE flag and skips setting the
caching flags.
But looking at perfmon counters like Cache: CopyRead and
Redirector:ReadBytesPaging we can see that on XP and .NET the redirector is
caching SPARSE files. The values of those counters for a SPARSE file are
identical to those values for the same non-SPARSE files.
On W2K, those values are 0 for SPARSE files.
We can send you logs, but it might be easier for you to try it yourself.

Thanks, Sara

----- Original Message -----
From: “Neal Christiansen”
To: “File Systems Developers”
Sent: Thursday, February 07, 2002 5:15 PM
Subject: [ntfsd] RE: The redirector doesn’t cache SPARSE files

Sara,

This issue is not fixed in XP or .NET.

I was wrong when I earlier reported that there was an issue with CC and
sparse files. These types of files are not cached because of problems
handling out of disk space issues between the local and remote system.

Neal Christiansen

This posting is provided “AS IS” with no warranties, and confers no
rights.“

-----Original Message-----
From: Sara Abraham [mailto:xxxxx@veritas.com]
Sent: Wednesday, February 06, 2002 08:52 PM
To: File Systems Developers
Subject: [ntfsd] RE: The redirector doesn’t cache SPARSE files

Thanks Neal,

We verified that this problem is fixed in XP and .NET. As for W2K, do
you
think that you’ll be able to provide a fix in SP3, and if so, what are
the
release dates ?
You say that there was a problem with CC caching SPARSE files. Was it
necessary to disable both read and write caching ? If disabling write
caching is sufficient, than a simple fix that allows read caching for
SPARSE
files would be good enough for us for now.

Sara

----- Original Message -----
From: “Neal Christiansen”
To: “File Systems Developers”
Sent: Wednesday, February 06, 2002 6:12 PM
Subject: [ntfsd] RE: The redirector doesn’t cache SPARSE files

Sara,

You are correct, the redirector currently does not cache files with the
SPARSE attribute set. This was originally done because there was a
problem
with CC caching sparse files. We believe this issue has been resolved
but
we need to investigate and see if the redirector can be changed. Thanks
for
bringing this issue up.

As a side note, unlike the fastfat and cdfs samples, the redirector
sample
in the IFSKit does not match what is in the shipping product. It
contains a
subset of the actual functionality.

Neal Christiansen

This posting is provided “AS IS” with no warranties, and confers no
rights.”

-----Original Message-----
From: Sara Abraham [mailto:xxxxx@veritas.com]
Sent: Monday, February 04, 2002 06:08 PM
To: File Systems Developers
Subject: [ntfsd] The redirector doesn’t cache SPARSE files

We have noticed that on W2K/SP2 the redirector does not attempt to cache
if
a file is marked with FILE_ATTRIBUTE_SPARSE_FILE, even though an oplock
(batch or level 2) is granted. The file does not have to actually be
sparse,
we mark a regular file with the SPARSE attribute and the redirector
doesn’t
cache.

Here is what we see:

The routine MRxSmbSetSrvOpenFlags () is invoked after create is
complete, to
set the buffering/caching flags on a file, based on the oplock level
which
was granted. It’s defined as:
VOID
MRxSmbSetSrvOpenFlags (
PRX_CONTEXT RxContext,
RX_FILE_TYPE StorageType,
PMRX_SRV_OPEN SrvOpen,
PMRX_SMB_SRV_OPEN smbSrvOpen
)

The code that has been released with the IFSKIT doesn’t show any
dependency
on file attributes,
the disassembly, however, shows the following:

mrxsmb!MRxSmbSetSrvOpenFlags:
f31b3730 8b44240c mov eax,[esp+0xc]
f31b3734 8b4808 mov ecx,[eax+0x8]
f31b3737 83603800 and dword ptr [eax+0x38],0x0
f31b373b f6415902 test byte ptr [ecx+0x59],0x2 ds:0023:e23f0c01=00
f31b373f 750d jnz mrxsmb!MRxSmbSetSrvOpenFlags+0x44 (f31b374e)

eax points to MRX_SRV_OPEN, and ecx to MRX_FCB. The test statement tests
the
Attributes field for FILE_ATTRIBUTE_SPARSE_FILE, and does not set any of
the
caching/buffering flags if the file is marked as sparse.

We would appreciate your comments on this issue.

Sara
—
You are currently subscribed to ntfsd as: xxxxx@Windows.Microsoft.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: xxxxx@veritas.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: xxxxx@Windows.Microsoft.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: xxxxx@veritas.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

OSR_Community_User · February 9, 2002, 1:04am

Hi Neal,

We have traced through the XP redirector code, and we have more details on
what happens.
In both XP and W2K, MRxSmbSetSrvOpenFlags() checks for the SPARSE attribute,
and doesn’t set the buffering flags if the file is SPARSE. In XP, a
RxChangeBufferingState() causes the buffering flag to change to
FCB_STATE_READCACHEING_ENABLED | FCB_STATE_READBUFFERING_ENABLED.
That’s why we see read caching in XP and .NET.

Here are some windbg traces:
We copy a SPARSE file from server to client and put a bp on
rdbss!RxCommonRead. When the bp is hit for the first time we see the
following MRX_FCB:
e1163008 0288ec03 00020240 8290ee40 8290ee78
e1163018 00a00000 00000000 00a00000 00000000
e1163028 00a00000 00000000 8290eed8 e1163034
e1163038 e1163034 00000000 828882f0 e11631d0
e1163048 00000000 00000002 c0000000 00000001
e1163058 00000000 00000001 00000000 00000000
e1163068 00a00000 00000000 00000220 00000000
e1163078 e11632d0 e11632d0 00000001 00000000

FcbState is c0000000 and Attributes is 220 (ARCHIVE, SPARSE)

on the following read, we have the following MRX_FCB:
e1163008 0288ec03 00020240 8290ee40 8290ee78
e1163018 00a00000 00000000 00a00000 00000000
e1163028 00a00000 00000000 8290eed8 e1163034
e1163038 e1163034 00000000 828882f0 e11631d0
e1163048 00000000 00000003 c3040000 00000002
e1163058 00000000 00000002 00000000 00000000
e1163068 00a00000 00000000 00000220 00000000
e1163078 e11632d0 e114ce68 00000002 00000000

FcbState is now c3040000 (FCB_STATE_READCACHEING_ENABLED |
FCB_STATE_READBUFFERING_ENABLED ) , Attributes are still 220.
We issued a ‘ba w’ to see who changed FcbState, we found that it was because
of a ChangeBufferingStateRequest.

Filemon on the server shows identical oplock behavior for W2K, XP and .NET:

22 7:51:31 PM System:4 FSCTL_REQUEST_BATCH_OPLOCK
D:\spdtest\spar
se SUCCESS
23 7:51:31 PM System:4 FASTIO_QUERY_NETWORK_OPEN_INFO
D:\spdtest\spar
se SUCCESS
24 7:51:31 PM System:4 IRP_MJ_QUERY_INFORMATION
D:\spdtest\spar
se SUCCESS FileEaInformation
25 7:51:31 PM System:4 IRP_MJ_QUERY_INFORMATION
D:\spdtest\spar
se SUCCESS FileStreamInformation
26 7:51:31 PM System:4 IRP_MJ_QUERY_INFORMATION
D:\spdtest\spar
se SUCCESS FileObjectIdInformation
27 7:51:31 PM System:4 IRP_MJ_QUERY_SECURITY
D:\spdtest\sparse
BUFFER OVERFLOW
28 7:51:31 PM System:4 IRP_MJ_QUERY_SECURITY
D:\spdtest\sparse
SUCCESS
29 7:51:31 PM System:4 IRP_MJ_QUERY_INFORMATION
D:\spdtest\spar
se SUCCESS FileInternalInformation
30 7:51:31 PM System:4 FASTIO_READ D:\spdtest\sparse
FAILURE
Offset: 0 Length: 512
31 7:51:31 PM System:4 IRP_MJ_READ D:\spdtest\sparse
SUCCESS
Offset: 0 Length: 512
32 7:51:31 PM System:4 IRP_MJ_CLOSE D:\spdtest\sparse
SUCCESS

33 7:51:31 PM System:4 IRP_MJ_CREATE D:\spdtest\sparse
OPLOCK
BREAK Attributes: Any Options: Open
34 7:51:31 PM System:4 FSCTL_OPLOCK_BREAK_NOTIFY
D:\spdtest\spar
se SUCCESS
35 7:51:31 PM System:4 FSCTL_OPLOCK_BREAK_ACKNOWLEDGE
D:\spdtest\spar
se SUCCESS
36 7:51:31 PM System:4 FSCTL_REQUEST_BATCH_OPLOCK
D:\spdtest\spar
se OPLOCK NOT GRANTED
37 7:51:31 PM System:4 FSCTL_REQUEST_OPLOCK_LEVEL_2
D:\spdtest\spar
se SUCCESS

A batch oplock is first granted, then broken and a level 2 oplock is
granted. I believe that this level 2 causes the ChangeBufferingState, which
on XP sets the read caching, but doesn’t do so on W2K.

So, you are caching reads of SPARSE files on W2K and XP. If it’s OK to cache
reads, why not change MRxSmbSetSrvOpenFlags() to only disable write caching,
and provide us with a simple fix for sp3.

Sara

----- Original Message -----
From: “Neal Christiansen”
To: “File Systems Developers”
Sent: Thursday, February 07, 2002 5:15 PM
Subject: [ntfsd] RE: The redirector doesn’t cache SPARSE files

Sara,

This issue is not fixed in XP or .NET.

I was wrong when I earlier reported that there was an issue with CC and
sparse files. These types of files are not cached because of problems
handling out of disk space issues between the local and remote system.

Neal Christiansen

This posting is provided “AS IS” with no warranties, and confers no
rights.“

-----Original Message-----
From: Sara Abraham [mailto:xxxxx@veritas.com]
Sent: Wednesday, February 06, 2002 08:52 PM
To: File Systems Developers
Subject: [ntfsd] RE: The redirector doesn’t cache SPARSE files

Thanks Neal,

We verified that this problem is fixed in XP and .NET. As for W2K, do
you
think that you’ll be able to provide a fix in SP3, and if so, what are
the
release dates ?
You say that there was a problem with CC caching SPARSE files. Was it
necessary to disable both read and write caching ? If disabling write
caching is sufficient, than a simple fix that allows read caching for
SPARSE
files would be good enough for us for now.

Sara

----- Original Message -----
From: “Neal Christiansen”
To: “File Systems Developers”
Sent: Wednesday, February 06, 2002 6:12 PM
Subject: [ntfsd] RE: The redirector doesn’t cache SPARSE files

Sara,

You are correct, the redirector currently does not cache files with the
SPARSE attribute set. This was originally done because there was a
problem
with CC caching sparse files. We believe this issue has been resolved
but
we need to investigate and see if the redirector can be changed. Thanks
for
bringing this issue up.

As a side note, unlike the fastfat and cdfs samples, the redirector
sample
in the IFSKit does not match what is in the shipping product. It
contains a
subset of the actual functionality.

Neal Christiansen

This posting is provided “AS IS” with no warranties, and confers no
rights.”

-----Original Message-----
From: Sara Abraham [mailto:xxxxx@veritas.com]
Sent: Monday, February 04, 2002 06:08 PM
To: File Systems Developers
Subject: [ntfsd] The redirector doesn’t cache SPARSE files

We have noticed that on W2K/SP2 the redirector does not attempt to cache
if
a file is marked with FILE_ATTRIBUTE_SPARSE_FILE, even though an oplock
(batch or level 2) is granted. The file does not have to actually be
sparse,
we mark a regular file with the SPARSE attribute and the redirector
doesn’t
cache.

Here is what we see:

The routine MRxSmbSetSrvOpenFlags () is invoked after create is
complete, to
set the buffering/caching flags on a file, based on the oplock level
which
was granted. It’s defined as:
VOID
MRxSmbSetSrvOpenFlags (
PRX_CONTEXT RxContext,
RX_FILE_TYPE StorageType,
PMRX_SRV_OPEN SrvOpen,
PMRX_SMB_SRV_OPEN smbSrvOpen
)

The code that has been released with the IFSKIT doesn’t show any
dependency
on file attributes,
the disassembly, however, shows the following:

mrxsmb!MRxSmbSetSrvOpenFlags:
f31b3730 8b44240c mov eax,[esp+0xc]
f31b3734 8b4808 mov ecx,[eax+0x8]
f31b3737 83603800 and dword ptr [eax+0x38],0x0
f31b373b f6415902 test byte ptr [ecx+0x59],0x2 ds:0023:e23f0c01=00
f31b373f 750d jnz mrxsmb!MRxSmbSetSrvOpenFlags+0x44 (f31b374e)

eax points to MRX_SRV_OPEN, and ecx to MRX_FCB. The test statement tests
the
Attributes field for FILE_ATTRIBUTE_SPARSE_FILE, and does not set any of
the
caching/buffering flags if the file is marked as sparse.

We would appreciate your comments on this issue.

Sara
—
You are currently subscribed to ntfsd as: xxxxx@Windows.Microsoft.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: xxxxx@veritas.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: xxxxx@Windows.Microsoft.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: xxxxx@veritas.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

OSR_Community_User · February 11, 2002, 11:45pm

Sara,

Thank you for the information. This is actually a bug and will be fixed
for the Server .net release (which is probably not what you wanted to
hear).

Neal Christiansen
Microsoft

This posting is provided “AS IS” with no warranties, and confers no
rights.

-----Original Message-----
From: Sara Abraham [mailto:xxxxx@veritas.com]
Sent: Friday, February 08, 2002 10:12 PM
To: File Systems Developers
Subject: [ntfsd] RE: The redirector doesn’t cache SPARSE files

Hi Neal,

We have traced through the XP redirector code, and we have more details
on
what happens.
In both XP and W2K, MRxSmbSetSrvOpenFlags() checks for the SPARSE
attribute,
and doesn’t set the buffering flags if the file is SPARSE. In XP, a
RxChangeBufferingState() causes the buffering flag to change to
FCB_STATE_READCACHEING_ENABLED | FCB_STATE_READBUFFERING_ENABLED.
That’s why we see read caching in XP and .NET.

Here are some windbg traces:
We copy a SPARSE file from server to client and put a bp on
rdbss!RxCommonRead. When the bp is hit for the first time we see the
following MRX_FCB:
e1163008 0288ec03 00020240 8290ee40 8290ee78
e1163018 00a00000 00000000 00a00000 00000000
e1163028 00a00000 00000000 8290eed8 e1163034
e1163038 e1163034 00000000 828882f0 e11631d0
e1163048 00000000 00000002 c0000000 00000001
e1163058 00000000 00000001 00000000 00000000
e1163068 00a00000 00000000 00000220 00000000
e1163078 e11632d0 e11632d0 00000001 00000000

FcbState is c0000000 and Attributes is 220 (ARCHIVE, SPARSE)

on the following read, we have the following MRX_FCB:
e1163008 0288ec03 00020240 8290ee40 8290ee78
e1163018 00a00000 00000000 00a00000 00000000
e1163028 00a00000 00000000 8290eed8 e1163034
e1163038 e1163034 00000000 828882f0 e11631d0
e1163048 00000000 00000003 c3040000 00000002
e1163058 00000000 00000002 00000000 00000000
e1163068 00a00000 00000000 00000220 00000000
e1163078 e11632d0 e114ce68 00000002 00000000

FcbState is now c3040000 (FCB_STATE_READCACHEING_ENABLED |
FCB_STATE_READBUFFERING_ENABLED ) , Attributes are still 220.
We issued a ‘ba w’ to see who changed FcbState, we found that it was
because
of a ChangeBufferingStateRequest.

Filemon on the server shows identical oplock behavior for W2K, XP and
.NET:

22 7:51:31 PM System:4 FSCTL_REQUEST_BATCH_OPLOCK
D:\spdtest\spar
se SUCCESS
23 7:51:31 PM System:4 FASTIO_QUERY_NETWORK_OPEN_INFO
D:\spdtest\spar
se SUCCESS
24 7:51:31 PM System:4 IRP_MJ_QUERY_INFORMATION
D:\spdtest\spar
se SUCCESS FileEaInformation
25 7:51:31 PM System:4 IRP_MJ_QUERY_INFORMATION
D:\spdtest\spar
se SUCCESS FileStreamInformation
26 7:51:31 PM System:4 IRP_MJ_QUERY_INFORMATION
D:\spdtest\spar
se SUCCESS FileObjectIdInformation
27 7:51:31 PM System:4 IRP_MJ_QUERY_SECURITY
D:\spdtest\sparse
BUFFER OVERFLOW
28 7:51:31 PM System:4 IRP_MJ_QUERY_SECURITY
D:\spdtest\sparse
SUCCESS
29 7:51:31 PM System:4 IRP_MJ_QUERY_INFORMATION
D:\spdtest\spar
se SUCCESS FileInternalInformation
30 7:51:31 PM System:4 FASTIO_READ
D:\spdtest\sparse
FAILURE
Offset: 0 Length: 512
31 7:51:31 PM System:4 IRP_MJ_READ
D:\spdtest\sparse
SUCCESS
Offset: 0 Length: 512
32 7:51:31 PM System:4 IRP_MJ_CLOSE
D:\spdtest\sparse
SUCCESS

33 7:51:31 PM System:4 IRP_MJ_CREATE
D:\spdtest\sparse
OPLOCK
BREAK Attributes: Any Options: Open
34 7:51:31 PM System:4 FSCTL_OPLOCK_BREAK_NOTIFY
D:\spdtest\spar
se SUCCESS
35 7:51:31 PM System:4 FSCTL_OPLOCK_BREAK_ACKNOWLEDGE
D:\spdtest\spar
se SUCCESS
36 7:51:31 PM System:4 FSCTL_REQUEST_BATCH_OPLOCK
D:\spdtest\spar
se OPLOCK NOT GRANTED
37 7:51:31 PM System:4 FSCTL_REQUEST_OPLOCK_LEVEL_2
D:\spdtest\spar
se SUCCESS

A batch oplock is first granted, then broken and a level 2 oplock is
granted. I believe that this level 2 causes the ChangeBufferingState,
which
on XP sets the read caching, but doesn’t do so on W2K.

So, you are caching reads of SPARSE files on W2K and XP. If it’s OK to
cache
reads, why not change MRxSmbSetSrvOpenFlags() to only disable write
caching,
and provide us with a simple fix for sp3.

Sara

----- Original Message -----
From: “Neal Christiansen”
To: “File Systems Developers”
Sent: Thursday, February 07, 2002 5:15 PM
Subject: [ntfsd] RE: The redirector doesn’t cache SPARSE files

Sara,

This issue is not fixed in XP or .NET.

I was wrong when I earlier reported that there was an issue with CC and
sparse files. These types of files are not cached because of problems
handling out of disk space issues between the local and remote system.

Neal Christiansen

This posting is provided “AS IS” with no warranties, and confers no
rights.“

-----Original Message-----
From: Sara Abraham [mailto:xxxxx@veritas.com]
Sent: Wednesday, February 06, 2002 08:52 PM
To: File Systems Developers
Subject: [ntfsd] RE: The redirector doesn’t cache SPARSE files

Thanks Neal,

We verified that this problem is fixed in XP and .NET. As for W2K, do
you
think that you’ll be able to provide a fix in SP3, and if so, what are
the
release dates ?
You say that there was a problem with CC caching SPARSE files. Was it
necessary to disable both read and write caching ? If disabling write
caching is sufficient, than a simple fix that allows read caching for
SPARSE
files would be good enough for us for now.

Sara

----- Original Message -----
From: “Neal Christiansen”
To: “File Systems Developers”
Sent: Wednesday, February 06, 2002 6:12 PM
Subject: [ntfsd] RE: The redirector doesn’t cache SPARSE files

Sara,

You are correct, the redirector currently does not cache files with the
SPARSE attribute set. This was originally done because there was a
problem
with CC caching sparse files. We believe this issue has been resolved
but
we need to investigate and see if the redirector can be changed. Thanks
for
bringing this issue up.

As a side note, unlike the fastfat and cdfs samples, the redirector
sample
in the IFSKit does not match what is in the shipping product. It
contains a
subset of the actual functionality.

Neal Christiansen

This posting is provided “AS IS” with no warranties, and confers no
rights.”

-----Original Message-----
From: Sara Abraham [mailto:xxxxx@veritas.com]
Sent: Monday, February 04, 2002 06:08 PM
To: File Systems Developers
Subject: [ntfsd] The redirector doesn’t cache SPARSE files

We have noticed that on W2K/SP2 the redirector does not attempt to cache
if
a file is marked with FILE_ATTRIBUTE_SPARSE_FILE, even though an oplock
(batch or level 2) is granted. The file does not have to actually be
sparse,
we mark a regular file with the SPARSE attribute and the redirector
doesn’t
cache.

Here is what we see:

The routine MRxSmbSetSrvOpenFlags () is invoked after create is
complete, to
set the buffering/caching flags on a file, based on the oplock level
which
was granted. It’s defined as:
VOID
MRxSmbSetSrvOpenFlags (
PRX_CONTEXT RxContext,
RX_FILE_TYPE StorageType,
PMRX_SRV_OPEN SrvOpen,
PMRX_SMB_SRV_OPEN smbSrvOpen
)

The code that has been released with the IFSKIT doesn’t show any
dependency
on file attributes,
the disassembly, however, shows the following:

mrxsmb!MRxSmbSetSrvOpenFlags:
f31b3730 8b44240c mov eax,[esp+0xc]
f31b3734 8b4808 mov ecx,[eax+0x8]
f31b3737 83603800 and dword ptr [eax+0x38],0x0
f31b373b f6415902 test byte ptr [ecx+0x59],0x2 ds:0023:e23f0c01=00
f31b373f 750d jnz mrxsmb!MRxSmbSetSrvOpenFlags+0x44 (f31b374e)

eax points to MRX_SRV_OPEN, and ecx to MRX_FCB. The test statement tests
the
Attributes field for FILE_ATTRIBUTE_SPARSE_FILE, and does not set any of
the
caching/buffering flags if the file is marked as sparse.

We would appreciate your comments on this issue.

Sara
—
You are currently subscribed to ntfsd as: xxxxx@Windows.Microsoft.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: xxxxx@veritas.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: xxxxx@Windows.Microsoft.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: xxxxx@veritas.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: xxxxx@Windows.Microsoft.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

The redirector doesn't cache SPARSE files

eax points to MRX_SRV_OPEN, and ecx to MRX_FCB. The test statement tests the Attributes field for FILE_ATTRIBUTE_SPARSE_FILE, and does not set any of the caching/buffering flags if the file is marked as sparse. ? We would appreciate your comments on this issue. ? Sara

Sara

eax points to MRX_SRV_OPEN, and ecx to MRX_FCB. The test statement tests the Attributes field for FILE_ATTRIBUTE_SPARSE_FILE, and does not set any of the caching/buffering flags if the file is marked as sparse.
?
We would appreciate your comments on this issue.
?
Sara