Hi,
I have a DECOMPRESSION filesystem filter driver, which is based on sfilter. The filter works with CDFS only, compression is performed by other means, the purpose of my filter to decompress files on the fly.
As you probably guess I get a problem with substituting compressed size of a file to the uncompressed size of the file. I have read many threads about “how bad to change file size”, but I have had put a lot of efforts on the compression architecture and would rather try to get it working than redesigning the entire approach.
Currently
- I properly process IRP_MJ_QUERY_INFORMATION/IRP_MJ_DIRECTORY_CONTROL so the correct file size is returned.
- For testing purposes I fail FastIO
- I properly process paging and noncached IRP_MJ_READ
- I pass through all cached or non-paging IRP_MJ_READ and this leads to the problem because CDFS uses information from FCB to check for EOF, e.g. ignoring all my efforts in IRP_MJ_QUERY_INFORMATION. As a result the filter is not given even a chance to read and decompress data which lay beyond EOF of a real file
I traced CDFS down and found that in most cases it performs all checks against FCB->FileSize value, so I decided to change this and all corresponding values in my CreateCompletion routine:
pFCB = (FSRTL_COMMON_FCB_HEADER*)IrpSp->FileObject->FsContext;
pFCB->FileSize = pFCB->ValidDataLength = UnpackedSize;
pFCB->AllocationSize.QuadPart = ( (UnpackedSize.QuadPart / 2048) + (UnpackedSize.QuadPart % 2048 == 0 ? 0 : 1) ) * 2048; //align to sector size
After that I faced a problem that I cannot determine the compressed size of a file, because even CDFS does not know it any more :). And I need this information for the decompression routine, but I hardcoded some sizes inside the filter and everything seemed to started working fine, so I plan to implement the hash which will store the real size of a file.
So my question: can such a solution lead to problems? I suspect that they may appear with the cache manager as I do not know if the cache is allocated before I changed the sizes in FCB or afterwards. And probably the problems may appear inside CDFS itself, if for example it decides to read data for a cached IRP_MJ_READ bypassing cache.
Thanks,
Roman
Another approach I was thinking about is processing all IRP_MJ_READ (cached, non-cached, paging, non-paging) in my routine. But I think this approach will give me a headache with FastIO and I will loose all advances of cache manager
Thanks,
Roman
“Roman Kudinov” ???/??? ? ??? ???: news:xxxxx@ntfsd…
Hi,
I have a DECOMPRESSION filesystem filter driver, which is based on sfilter. The filter works with CDFS only, compression is performed by other means, the purpose of my filter to decompress files on the fly.
As you probably guess I get a problem with substituting compressed size of a file to the uncompressed size of the file. I have read many threads about “how bad to change file size”, but I have had put a lot of efforts on the compression architecture and would rather try to get it working than redesigning the entire approach.
Currently
- I properly process IRP_MJ_QUERY_INFORMATION/IRP_MJ_DIRECTORY_CONTROL so the correct file size is returned.
- For testing purposes I fail FastIO
- I properly process paging and noncached IRP_MJ_READ
- I pass through all cached or non-paging IRP_MJ_READ and this leads to the problem because CDFS uses information from FCB to check for EOF, e.g. ignoring all my efforts in IRP_MJ_QUERY_INFORMATION. As a result the filter is not given even a chance to read and decompress data which lay beyond EOF of a real file
I traced CDFS down and found that in most cases it performs all checks against FCB->FileSize value, so I decided to change this and all corresponding values in my CreateCompletion routine:
pFCB = (FSRTL_COMMON_FCB_HEADER*)IrpSp->FileObject->FsContext;
pFCB->FileSize = pFCB->ValidDataLength = UnpackedSize;
pFCB->AllocationSize.QuadPart = ( (UnpackedSize.QuadPart / 2048) + (UnpackedSize.QuadPart % 2048 == 0 ? 0 : 1) ) * 2048; //align to sector size
After that I faced a problem that I cannot determine the compressed size of a file, because even CDFS does not know it any more :). And I need this information for the decompression routine, but I hardcoded some sizes inside the filter and everything seemed to started working fine, so I plan to implement the hash which will store the real size of a file.
So my question: can such a solution lead to problems? I suspect that they may appear with the cache manager as I do not know if the cache is allocated before I changed the sizes in FCB or afterwards. And probably the problems may appear inside CDFS itself, if for example it decides to read data for a cached IRP_MJ_READ bypassing cache.
Thanks,
Roman
It seems this method works fine, at least I did not get any major troubles with it. The only problem I have at the moment is tracking of File Context, it looks like the method described in FAQ does not work properly.
“Roman Kudinov” ???/??? ? ??? ???: news:xxxxx@ntfsd…
Another approach I was thinking about is processing all IRP_MJ_READ (cached, non-cached, paging, non-paging) in my routine. But I think this approach will give me a headache with FastIO and I will loose all advances of cache manager
Thanks,
Roman
“Roman Kudinov” ???/??? ? ??? ???: news:xxxxx@ntfsd…
Hi,
I have a DECOMPRESSION filesystem filter driver, which is based on sfilter. The filter works with CDFS only, compression is performed by other means, the purpose of my filter to decompress files on the fly.
As you probably guess I get a problem with substituting compressed size of a file to the uncompressed size of the file. I have read many threads about “how bad to change file size”, but I have had put a lot of efforts on the compression architecture and would rather try to get it working than redesigning the entire approach.
Currently
- I properly process IRP_MJ_QUERY_INFORMATION/IRP_MJ_DIRECTORY_CONTROL so the correct file size is returned.
- For testing purposes I fail FastIO
- I properly process paging and noncached IRP_MJ_READ
- I pass through all cached or non-paging IRP_MJ_READ and this leads to the problem because CDFS uses information from FCB to check for EOF, e.g. ignoring all my efforts in IRP_MJ_QUERY_INFORMATION. As a result the filter is not given even a chance to read and decompress data which lay beyond EOF of a real file
I traced CDFS down and found that in most cases it performs all checks against FCB->FileSize value, so I decided to change this and all corresponding values in my CreateCompletion routine:
pFCB = (FSRTL_COMMON_FCB_HEADER*)IrpSp->FileObject->FsContext;
pFCB->FileSize = pFCB->ValidDataLength = UnpackedSize;
pFCB->AllocationSize.QuadPart = ( (UnpackedSize.QuadPart / 2048) + (UnpackedSize.QuadPart % 2048 == 0 ? 0 : 1) ) * 2048; //align to sector size
After that I faced a problem that I cannot determine the compressed size of a file, because even CDFS does not know it any more :). And I need this information for the decompression routine, but I hardcoded some sizes inside the filter and everything seemed to started working fine, so I plan to implement the hash which will store the real size of a file.
So my question: can such a solution lead to problems? I suspect that they may appear with the cache manager as I do not know if the cache is allocated before I changed the sizes in FCB or afterwards. And probably the problems may appear inside CDFS itself, if for example it decides to read data for a cached IRP_MJ_READ bypassing cache.
Thanks,
Roman
The FileContext tracking method in the FAQ works fine. There have been
hundreds of posts on it and it has been used in many implementations.
If you are manipulating the filesize within the Fcb, which you do not
own, then you will need to follow the same locking semantics as the
underlying filesystem for the Fcb to prevent corruption.
Pete
Kernel Drivers
Windows File System and Device Driver Consulting
www.KernelDrivers.com
(303)546-0300
Roman Kudinov wrote:
It seems this method works fine, at least I did not get any major
troubles with it. The only problem I have at the moment is tracking of
File Context, it looks like the method described in FAQ does not work
properly.
“Roman Kudinov” >
> ???/??? ? ??? ???: news:xxxxx@ntfsd…
> Another approach I was thinking about is processing all IRP_MJ_READ
> (cached, non-cached, paging, non-paging) in my routine. But I think
> this approach will give me a headache with FastIO and I will loose
> all advances of cache manager
>
>
> Thanks,
> Roman
>
>
>
> “Roman Kudinov” >
> ???/??? ? ??? ???: news:xxxxx@ntfsd…
> Hi,
>
> I have a DECOMPRESSION filesystem filter driver, which is based
> on sfilter. The filter works with CDFS only, compression is
> performed by other means, the purpose of my filter to decompress
> files on the fly.
>
> As you probably guess I get a problem with substituting
> compressed size of a file to the uncompressed size of the file.
> I have read many threads about “how bad to change file size”,
> but I have had put a lot of efforts on the compression
> architecture and would rather try to get it working than
> redesigning the entire approach.
>
>
> Currently
> - I properly process
> IRP_MJ_QUERY_INFORMATION/IRP_MJ_DIRECTORY_CONTROL so the correct
> file size is returned.
> - For testing purposes I fail FastIO
> - I properly process paging and noncached IRP_MJ_READ
> - I pass through all cached or non-paging IRP_MJ_READ and this
> leads to the problem because CDFS uses information from FCB to
> check for EOF, e.g. ignoring all my efforts in
> IRP_MJ_QUERY_INFORMATION. As a result the filter is not given
> even a chance to read and decompress data which lay beyond EOF
> of a real file
>
> I traced CDFS down and found that in most cases it performs all
> checks against FCB->FileSize value, so I decided to change this
> and all corresponding values in my CreateCompletion routine:
>
> pFCB = (FSRTL_COMMON_FCB_HEADER*)IrpSp->FileObject->FsContext;
> pFCB->FileSize = pFCB->ValidDataLength = UnpackedSize;
> pFCB->AllocationSize.QuadPart = ( (UnpackedSize.QuadPart /
> 2048) + (UnpackedSize.QuadPart % 2048 == 0 ? 0 : 1) ) * 2048;
> //align to sector size
>
> After that I faced a problem that I cannot determine the
> compressed size of a file, because even CDFS does not know it
> any more :). And I need this information for the decompression
> routine, but I hardcoded some sizes inside the filter and
> everything seemed to started working fine, so I plan to
> implement the hash which will store the real size of a file.
>
>
> So my question: can such a solution lead to problems? I suspect
> that they may appear with the cache manager as I do not know if
> the cache is allocated before I changed the sizes in FCB or
> afterwards. And probably the problems may appear inside CDFS
> itself, if for example it decides to read data for a cached
> IRP_MJ_READ bypassing cache.
>
>
> Thanks,
> Roman
>
>
> —
> Questions? First check the IFS FAQ at
> https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
> To unsubscribe send a blank email to xxxxx@lists.osr.com
Hi Peter,
Probably I used the wrong words, by “not properly” I mean that it works not
like I expected it to.
In some time after I remove FsContext from hash in IRP_MJ_CLOSE dispatch
(using the method described in FAQ), I get exactly the same FsContext value
for exactly the same file in IRP_MJ_CREATE completion routine. Thus I
suspect that FCB is not deallocated by FS right after a file is completely
dereferenced (or I do something wrong). And this involves some headache
because I get FCB with the file size modified by me previously. So I have to
find a way to get to know if I get already modified FCB or the same address
is reused for another file.
Currently I have implemented the rest of the filter the way which just
ignores this headache, but in the case I track FsContext not correctly this
may lead to big problems. FAQ states that the proposed tracking will work
for read/write only, so probably I just faced the limitations of the
tracking algorithm.
Thanks for your suggestion about locking.
–
Roman
“Peter Scott” ÓÏÏÂÝÉÌ/ÓÏÏÂÝÉÌÁ × ÎÏ×ÏÓÔÑÈ
ÓÌÅÄÕÀÝÅÅ: news:xxxxx@ntfsd…
The FileContext tracking method in the FAQ works fine. There have been
hundreds of posts on it and it has been used in many implementations.
If you are manipulating the filesize within the Fcb, which you do not
own, then you will need to follow the same locking semantics as the
underlying filesystem for the Fcb to prevent corruption.
Pete
Kernel Drivers
Windows File System and Device Driver Consulting
www.KernelDrivers.com
(303)546-0300
Roman Kudinov wrote:
> It seems this method works fine, at least I did not get any major troubles
> with it. The only problem I have at the moment is tracking of File
> Context, it looks like the method described in FAQ does not work properly.
>
> “Roman Kudinov” >
> ÓÏÏÂÝÉÌ/ÓÏÏÂÝÉÌÁ × ÎÏ×ÏÓÔÑÈ ÓÌÅÄÕÀÝÅÅ: news:xxxxx@ntfsd…
> Another approach I was thinking about is processing all IRP_MJ_READ
> (cached, non-cached, paging, non-paging) in my routine. But I think
> this approach will give me a headache with FastIO and I will loose
> all advances of cache manager
> Thanks,
> Roman
>
> “Roman Kudinov” >
> ÓÏÏÂÝÉÌ/ÓÏÏÂÝÉÌÁ × ÎÏ×ÏÓÔÑÈ ÓÌÅÄÕÀÝÅÅ: news:xxxxx@ntfsd…
> Hi,
> I have a DECOMPRESSION filesystem filter driver, which is based
> on sfilter. The filter works with CDFS only, compression is
> performed by other means, the purpose of my filter to decompress
> files on the fly.
> As you probably guess I get a problem with substituting
> compressed size of a file to the uncompressed size of the file.
> I have read many threads about “how bad to change file size”,
> but I have had put a lot of efforts on the compression
> architecture and would rather try to get it working than
> redesigning the entire approach.
> Currently
> - I properly process
> IRP_MJ_QUERY_INFORMATION/IRP_MJ_DIRECTORY_CONTROL so the correct
> file size is returned.
> - For testing purposes I fail FastIO
> - I properly process paging and noncached IRP_MJ_READ
> - I pass through all cached or non-paging IRP_MJ_READ and this
> leads to the problem because CDFS uses information from FCB to
> check for EOF, e.g. ignoring all my efforts in
> IRP_MJ_QUERY_INFORMATION. As a result the filter is not given
> even a chance to read and decompress data which lay beyond EOF
> of a real file
> I traced CDFS down and found that in most cases it performs all
> checks against FCB->FileSize value, so I decided to change this
> and all corresponding values in my CreateCompletion routine:
> pFCB = (FSRTL_COMMON_FCB_HEADER*)IrpSp->FileObject->FsContext;
> pFCB->FileSize = pFCB->ValidDataLength = UnpackedSize;
> pFCB->AllocationSize.QuadPart = ( (UnpackedSize.QuadPart /
> 2048) + (UnpackedSize.QuadPart % 2048 == 0 ? 0 : 1) ) * 2048;
> //align to sector size
> After that I faced a problem that I cannot determine the
> compressed size of a file, because even CDFS does not know it
> any more :). And I need this information for the decompression
> routine, but I hardcoded some sizes inside the filter and
> everything seemed to started working fine, so I plan to
> implement the hash which will store the real size of a file.
> So my question: can such a solution lead to problems? I suspect
> that they may appear with the cache manager as I do not know if
> the cache is allocated before I changed the sizes in FCB or
> afterwards. And probably the problems may appear inside CDFS
> itself, if for example it decides to read data for a cached
> IRP_MJ_READ bypassing cache.
> Thanks,
> Roman
>
>
> —
> Questions? First check the IFS FAQ at
> https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
> To unsubscribe send a blank email to xxxxx@lists.osr.com