PFN list corrupt - copying file in FS filter

NTFSD
1

I’m working on a file system filter, and one of its tasks is to copy a file
just prior to deletion. The filter catches the FileDispositionInformation
FileInformationClass of the IRP_MJ_SET_INFORMATION IRP.

I open the destination file with the ZwCreateFile routine, and use the
ZwWrite routine to write data into the destination file. Since the file to
be copied is already open, I roll my own IRPs to read data out of the file
using IoBuildAsynchronousFsdRequest(). I’m using a 16K non-paged buffer to
read and write the data.

When deleting a large file (5 MB), the copy routine got about 800K copied
before hitting the following bugcheck (0x4E is PFN_LIST_CORRUPT):

*** Fatal System Error: 0x0000004e
(0x00000007,0x00001BE0,0x00000001,0x00000000)

The stack looks like:

ChildEBP RetAddr Args to Child
f5436d48 8042c068 00000003 00001be0 81464d00
ntoskrnl!DbgBreakPointWithStatus+0x4
f54370d0 80449cbb 00000000 00000007 00001be0 ntoskrnl!KeBugCheckEx0x154
8129aaa4 0000015e ffffffff ffffffff ffffffff
ntoskrnl!MmTrimAllSystemPagableMemory+0x6c8a

I suspect something about the buffer that I’m using. I previously checked
the irp->MdlAddress and the AssociatedIrp->SystemBuffer addresses in the IRP
created by IoBuildAsynchronousFsdRequest and they were NULL. I concluded
(perhaps incorrectly) that since I was using a non-paged buffer, it was fine
for these addresses to be NULL. Does anyone have any ideas about what I
might be doing incorrectly?

Brad

You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

2

I had the problem similar to yours.
I could solve my problem by freeing MDLs allocated by the FSD below
in my completion routine. I didn’t know the FSD below sometimes
allocates the MDLs when I roll my own IRPs.
My code is as follows.

if(pIRP->MdlAddress){
pMdl = pIRP->MdlAddress;
while(pMdl){
pNextMdl = pMdl->Next;
MmUnlockPages(pMdl);
IoFreeMdl(pMdl);
pMdl = pNextMdl;
}
}

I hope this helps.

Regards,

Takashi

You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

3

I saw similar code in the archives and put the following code in place. The
algorithm is slightly different than yours and I’m wondering if it might
make a difference. While your algorithm calls IoFreeMdl for each MDL, the
algorithm I’m utilizing calls IoFreeMdl just once. Here’s the algorithm:

pMdl = irp->MdlAddress;
if (NULL != pMdl)
{
do
{
MmUnlockPages(pMdl);
pMdl = pMdl->Next;
} while (pMdl != NULL);
IoFreeMdl(irp->MdlAddress);
}

Should IoFreeMdl be called for each pMdl or just the irp->MdlAddress?

I find it interesting that the underlying filesystem has the ‘freedom’ to
use the Mdl, and wonder why it would do so. I’ve copied large 1 amd 5 MB
files (using a 16K non-paged buffer) and found that -a single- call to the
underlying filesystem decided to allocate an MDL. And in all of those cases,
pMdl->Next was NULL (so perhaps I have yet another bug in waiting). Can
someone provide some insight as to why the filesystem would choose to use an
MDL?

Brad

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of
xxxxx@itg.hitachi.co.jp
Sent: Wednesday, February 21, 2001 9:11 PM
To: File Systems Developers
Subject: [ntfsd] Re: PFN list corrupt - copying file in FS filter

I had the problem similar to yours.
I could solve my problem by freeing MDLs allocated by the FSD below
in my completion routine. I didn’t know the FSD below sometimes
allocates the MDLs when I roll my own IRPs.
My code is as follows.

if(pIRP->MdlAddress){
pMdl = pIRP->MdlAddress;
while(pMdl){
pNextMdl = pMdl->Next;
MmUnlockPages(pMdl);
IoFreeMdl(pMdl);
pMdl = pNextMdl;
}
}

I hope this helps.

Regards,

Takashi

You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

4

Hi,

I found my code in the fastfat sample code in the IFS kit.
I do not know which algorithm is right, yours or mine.
Is there anybody that can give good explanation about this?

Regards,

Takashi

I saw similar code in the archives and put the following code in place. The
algorithm is slightly different than yours and I’m wondering if it might
make a difference. While your algorithm calls IoFreeMdl for each MDL, the
algorithm I’m utilizing calls IoFreeMdl just once. Here’s the algorithm:

pMdl = irp->MdlAddress;
if (NULL != pMdl)
{
do
{
MmUnlockPages(pMdl);
pMdl = pMdl->Next;
} while (pMdl != NULL);
IoFreeMdl(irp->MdlAddress);
}

Should IoFreeMdl be called for each pMdl or just the irp->MdlAddress?

I find it interesting that the underlying filesystem has the ‘freedom’ to
use the Mdl, and wonder why it would do so. I’ve copied large 1 amd 5 MB
files (using a 16K non-paged buffer) and found that -a single- call to the
underlying filesystem decided to allocate an MDL. And in all of those cases,
pMdl->Next was NULL (so perhaps I have yet another bug in waiting). Can
someone provide some insight as to why the filesystem would choose to use an
MDL?

Brad

> -----Original Message-----
> From: xxxxx@lists.osr.com
> Sent: Wednesday, February 21, 2001 9:11 PM
> Subject: [ntfsd] Re: PFN list corrupt - copying file in FS filter
>
> I had the problem similar to yours.
> I could solve my problem by freeing MDLs allocated by the FSD below
> in my completion routine. I didn’t know the FSD below sometimes
> allocates the MDLs when I roll my own IRPs.
> My code is as follows.
>
> if(pIRP->MdlAddress){
> pMdl = pIRP->MdlAddress;
> while(pMdl){
> pNextMdl = pMdl->Next;
> MmUnlockPages(pMdl);
> IoFreeMdl(pMdl);
> pMdl = pNextMdl;
> }
> }
>
> I hope this helps.
>
> Regards,
>
> Takashi

You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

5

IoFreeMdl() does not walk the list of chained MDLs, so the first
description in this thread is accurate. The second (Brad’s) will leak
the memory for each MDL chained off of the first. Chaining happens when
you IoAllocateMdl with SecondaryBuffer=TRUE and supply an IRP, which
must already have an MDL associated with it.

MDLs are physical descriptions of pinned buffers - among other reasons,
if the request is being posted out of the context of the original
thread, you must build one so that the receiving thread knows what
memory is associated with the request. In the IFS Kit, read
FatPrePostIrp. This should make sense to you now.

The chaining behavior would be a great thing to clarify in the DDK
documentation.

-----Original Message-----
From: xxxxx@itg.hitachi.co.jp [mailto:xxxxx@itg.hitachi.co.jp]
Sent: Wednesday, February 21, 2001 5:01 PM
To: File Systems Developers
Subject: [ntfsd] Re: PFN list corrupt - copying file in FS filter

Hi,

I found my code in the fastfat sample code in the IFS kit.
I do not know which algorithm is right, yours or mine.
Is there anybody that can give good explanation about this?

Regards,

Takashi

I saw similar code in the archives and put the following code in
place. The
algorithm is slightly different than yours and I’m wondering if it
might
make a difference. While your algorithm calls IoFreeMdl for each MDL,
the
algorithm I’m utilizing calls IoFreeMdl just once. Here’s the
algorithm:

pMdl = irp->MdlAddress;
if (NULL != pMdl)
{
do
{
MmUnlockPages(pMdl);
pMdl = pMdl->Next;
} while (pMdl != NULL);
IoFreeMdl(irp->MdlAddress);
}

Should IoFreeMdl be called for each pMdl or just the irp->MdlAddress?

I find it interesting that the underlying filesystem has the ‘freedom’
to
use the Mdl, and wonder why it would do so. I’ve copied large 1 amd 5
MB
files (using a 16K non-paged buffer) and found that -a single- call to
the
underlying filesystem decided to allocate an MDL. And in all of those
cases,
pMdl->Next was NULL (so perhaps I have yet another bug in waiting).
Can
someone provide some insight as to why the filesystem would choose to
use an
MDL?

Brad

> -----Original Message-----
> From: xxxxx@lists.osr.com
> Sent: Wednesday, February 21, 2001 9:11 PM
> Subject: [ntfsd] Re: PFN list corrupt - copying file in FS filter
>
> I had the problem similar to yours.
> I could solve my problem by freeing MDLs allocated by the FSD below
> in my completion routine. I didn’t know the FSD below sometimes
> allocates the MDLs when I roll my own IRPs.
> My code is as follows.
>
> if(pIRP->MdlAddress){
> pMdl = pIRP->MdlAddress;
> while(pMdl){
> pNextMdl = pMdl->Next;
> MmUnlockPages(pMdl);
> IoFreeMdl(pMdl);
> pMdl = pNextMdl;
> }
> }
>
> I hope this helps.
>
> Regards,
>
> Takashi

You are currently subscribed to ntfsd as: xxxxx@exchange.microsoft.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

6

Thank you for the explanation Dan. Adding the MDL freeing code (even the my
incorrect initial implementation) did clear up the problem with the corrupt
PFN list. However, it is reassuring that the second implementation will
avoid the memory leak you point out.

Brad

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of Daniel Lovinger
Sent: Thursday, February 22, 2001 7:58 PM
To: File Systems Developers
Subject: [ntfsd] Re: PFN list corrupt - copying file in FS filter

IoFreeMdl() does not walk the list of chained MDLs, so the first
description in this thread is accurate. The second (Brad’s) will leak
the memory for each MDL chained off of the first. Chaining happens when
you IoAllocateMdl with SecondaryBuffer=TRUE and supply an IRP, which
must already have an MDL associated with it.

MDLs are physical descriptions of pinned buffers - among other reasons,
if the request is being posted out of the context of the original
thread, you must build one so that the receiving thread knows what
memory is associated with the request. In the IFS Kit, read
FatPrePostIrp. This should make sense to you now.

The chaining behavior would be a great thing to clarify in the DDK
documentation.

-----Original Message-----
From: xxxxx@itg.hitachi.co.jp [mailto:xxxxx@itg.hitachi.co.jp]
Sent: Wednesday, February 21, 2001 5:01 PM
To: File Systems Developers
Subject: [ntfsd] Re: PFN list corrupt - copying file in FS filter

Hi,

I found my code in the fastfat sample code in the IFS kit.
I do not know which algorithm is right, yours or mine.
Is there anybody that can give good explanation about this?

Regards,

Takashi

> I saw similar code in the archives and put the following code in
place. The
> algorithm is slightly different than yours and I’m wondering if it
might
> make a difference. While your algorithm calls IoFreeMdl for each MDL,
the
> algorithm I’m utilizing calls IoFreeMdl just once. Here’s the
algorithm:
>
> pMdl = irp->MdlAddress;
> if (NULL != pMdl)
> {
> do
> {
> MmUnlockPages(pMdl);
> pMdl = pMdl->Next;
> } while (pMdl != NULL);
> IoFreeMdl(irp->MdlAddress);
> }
>
> Should IoFreeMdl be called for each pMdl or just the irp->MdlAddress?
>
> I find it interesting that the underlying filesystem has the ‘freedom’
to
> use the Mdl, and wonder why it would do so. I’ve copied large 1 amd 5
MB
> files (using a 16K non-paged buffer) and found that -a single- call to
the
> underlying filesystem decided to allocate an MDL. And in all of those
cases,
> pMdl->Next was NULL (so perhaps I have yet another bug in waiting).
Can
> someone provide some insight as to why the filesystem would choose to
use an
> MDL?
>
> Brad
>
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > Sent: Wednesday, February 21, 2001 9:11 PM
> > Subject: [ntfsd] Re: PFN list corrupt - copying file in FS filter
> >
> > I had the problem similar to yours.
> > I could solve my problem by freeing MDLs allocated by the FSD below
> > in my completion routine. I didn’t know the FSD below sometimes
> > allocates the MDLs when I roll my own IRPs.
> > My code is as follows.
> >
> > if(pIRP->MdlAddress){
> > pMdl = pIRP->MdlAddress;
> > while(pMdl){
> > pNextMdl = pMdl->Next;
> > MmUnlockPages(pMdl);
> > IoFreeMdl(pMdl);
> > pMdl = pNextMdl;
> > }
> > }
> >
> > I hope this helps.
> >
> > Regards,
> >
> > Takashi

You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com