Cache purge cause KERNEL_APC_PENDING_DURING_EXIT during shutdown

NTFSD
1

Hi,
I bring a cache purge issue again even though it has been discussed
many times.

I have a filter driver to encrypt/decrypt on the fly. One of the
requirement is when each time application reads the encrypted file,
the filter driver need to force it not read from cache, read from disk
instead, so that driver can decrypt data. I use following code someone
suggested in this forum to purge cache in post IRP_MJ_CREATE (completion
routine) and force system to launch paging IO to read from disk.

pFCBHeader = (PFSRTL_COMMON_FCB_HEADER)(pIrpStack->FileObject->FsContext);
pSection = pIrpStack->FileObject->SectionObjectPointer;

KeEnterCriticalRegion();
ExAcquireResourceExclusiveLite(pFcbHeader->Resource, TRUE);
ExAcquireResourceExclusiveLite(pFcbHeader->PagingIoResource, TRUE);

CcFlushCache(pSection, 0, 0, NULL);

If (pSection->ImageSectionObject != NULL)
MmFlushImageSection(pSection, MmFlushForWrite);

If (pSection->DataSectionObject != NULL)
CcPurgeCacheSection(pSection, NULL, 0, TRUE);

CcUninitializeCacheMap(pIrpStack->FileObject, NULL, NULL);

ExReleaseResourceLite(pFcbHeader->PagingIoResource);
ExReleaseResourceLite(pFcbHeader->Resource);
KeLeaveCriticalRegion();

After the cache is purged, I can see IRP_MJ_CLEANUP, IRP_MJ_CLOSE
and IRP_MJ_READ for paging request. So, it seems cache is purged
successfully and filter driver can decrypt the file for paging IO.

However, on one of our XP SP2 NTFS testing machine(there are more
than 20 testing machines set to different configurations), it
cause KERNEL_APC_PENDING_DURING_EXIT on every time machine reboot
or shutdown. All other testing machines work fine.

I put the memory dump in here:
In memory dump, the arg2 is negative. I can not figure out how I call
KeEnterCriticalRegion() cause APC disable count is not zero. If I
disable all Cc and Mm functions and just leave resource acquire and
release in there, the problem won't happen. Of course, when application
second time open the encrypted file, it will read from cache, not from
disk because driver don't see paging IO. So, it seems that ntoskrnal.exe
is broken by I call Cc Mm functions. But I can't get reasonable
interpretation and don't know how to solve that.

Do you guys have any ideal? Any suggestion will be appreciated.
I'm sorry that is long post because I try to give some detail about what I
did and where the problem is.

BugCheck 20, {8a210cac, fffffffe, 0, 0}

Probably caused by : ntoskrnl.exe ( nt!PsWaitForAllProcesses+9e )

Followup: MachineOwner

kd> !analyze -v
****************************************************************
*
*
* Bugcheck Analysis
*
*
*
****************************************************************

KERNEL_APC_PENDING_DURING_EXIT (20)
Arguments:
Arg1: 8a210cac, The address of the APC found pending during exit.
Arg2: fffffffe, The thread's APC disable count
Arg3: 00000000, The current IRQL
Arg4: 00000000

Debugging Details:

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0x20

LAST_CONTROL_TRANSFER: from 805f9e4c to 8053331e

STACK_TEXT:
aed7ac50 805f9e4c 00000020 8a210cac fffffffe nt!KeBugCheckEx+0x1b
aed7ad08 8057a46a 00000000 8a14bbc0 00000000 nt!PsWaitForAllProcesses+0x9e
aed7ad28 80582d13 8a14bbc0 00000000 aed7ad64
nt!PspTerminateThreadByPointer+0x52
aed7ad54 804de7ec 00000000 00000000 00eaff24 nt!NtTerminateProcess+0x118
aed7ad54 7c90eb94 00000000 00000000 00eaff24 nt!KiFastCallEntry+0xf8
WARNING: Frame IP not in any known module. Following frames may be wrong.
00eafe24 00000000 00000000 00000000 00000000 0x7c90eb94

FOLLOWUP_IP:
nt!PsWaitForAllProcesses+9e
805f9e4c 43 inc ebx

SYMBOL_STACK_INDEX: 1

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: nt!PsWaitForAllProcesses+9e

MODULE_NAME: nt

IMAGE_NAME: ntoskrnl.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 42250ff9

STACK_COMMAND: kb

BUCKET_ID: 0x20_nt!PsWaitForAllProcesses+9e

Followup: MachineOwner

kd> kb
ChildEBP RetAddr Args to Child
aed7ac50 805f9e4c 00000020 8a210cac fffffffe nt!KeBugCheckEx+0x1b
aed7ad08 8057a46a 00000000 8a14bbc0 00000000 nt!PsWaitForAllProcesses+0x9e
aed7ad28 80582d13 8a14bbc0 00000000 aed7ad64
nt!PspTerminateThreadByPointer+0x52
aed7ad54 804de7ec 00000000 00000000 00eaff24 nt!NtTerminateProcess+0x118
aed7ad54 7c90eb94 00000000 00000000 00eaff24 nt!KiFastCallEntry+0xf8
WARNING: Frame IP not in any known module. Following frames may be wrong.
00eafe24 00000000 00000000 00000000 00000000 0x7c90eb94

2

Do you have structured exception handling set up in this driver? This
error (which is frustrating to track down) could occur (for example) if
outside the scope of the function for which you give a code fragment you
have something like:

__try {

// call function implementing the code you provided

} __except (EXCEPTION_EXECUTE_HANDLER) {

// handle exception, ignore, convert to error code.
}

Because my guess is that your Cc/Mm routines are raising an error (after
all, you can't do what you are trying to do here 100% of the time, and
it sounds like you've found one of those configurations that leads us to
tell people "you can't do it this way". But we can never seem to
convince anyone of this fact...)

If this is the case I'd suggest that you try putting a __try/__except
block around each of your Cc and Mm calls until you see which one is
raising an exception.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.

Looking forward to seeing you at the next OSR File Systems class in
Boston, MA April 18-21, 2006 (note new date - MS scheduled plugfest the
same week again.)

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of David Wu
Sent: Thursday, March 09, 2006 11:04 PM
To: ntfsd redirect
Subject: [ntfsd] Cache purge cause KERNEL_APC_PENDING_DURING_EXIT during
shutdown

Hi,
I bring a cache purge issue again even though it has been discussed
many times.

I have a filter driver to encrypt/decrypt on the fly. One of the
requirement is when each time application reads the encrypted file,
the filter driver need to force it not read from cache, read from disk
instead, so that driver can decrypt data. I use following code someone
suggested in this forum to purge cache in post IRP_MJ_CREATE (completion
routine) and force system to launch paging IO to read from disk.

pFCBHeader =
(PFSRTL_COMMON_FCB_HEADER)(pIrpStack->FileObject->FsContext);
pSection = pIrpStack->FileObject->SectionObjectPointer;

KeEnterCriticalRegion();
ExAcquireResourceExclusiveLite(pFcbHeader->Resource, TRUE);
ExAcquireResourceExclusiveLite(pFcbHeader->PagingIoResource, TRUE);

CcFlushCache(pSection, 0, 0, NULL);

If (pSection->ImageSectionObject != NULL)
MmFlushImageSection(pSection, MmFlushForWrite);

If (pSection->DataSectionObject != NULL)
CcPurgeCacheSection(pSection, NULL, 0, TRUE);

CcUninitializeCacheMap(pIrpStack->FileObject, NULL, NULL);

ExReleaseResourceLite(pFcbHeader->PagingIoResource);
ExReleaseResourceLite(pFcbHeader->Resource);
KeLeaveCriticalRegion();

After the cache is purged, I can see IRP_MJ_CLEANUP, IRP_MJ_CLOSE
and IRP_MJ_READ for paging request. So, it seems cache is purged
successfully and filter driver can decrypt the file for paging IO.

However, on one of our XP SP2 NTFS testing machine(there are more
than 20 testing machines set to different configurations), it
cause KERNEL_APC_PENDING_DURING_EXIT on every time machine reboot
or shutdown. All other testing machines work fine.

I put the memory dump in here:
In memory dump, the arg2 is negative. I can not figure out how I call
KeEnterCriticalRegion() cause APC disable count is not zero. If I
disable all Cc and Mm functions and just leave resource acquire and
release in there, the problem won't happen. Of course, when application
second time open the encrypted file, it will read from cache, not from
disk because driver don't see paging IO. So, it seems that ntoskrnal.exe
is broken by I call Cc Mm functions. But I can't get reasonable
interpretation and don't know how to solve that.

Do you guys have any ideal? Any suggestion will be appreciated.
I'm sorry that is long post because I try to give some detail about what
I
did and where the problem is.

BugCheck 20, {8a210cac, fffffffe, 0, 0}

Probably caused by : ntoskrnl.exe ( nt!PsWaitForAllProcesses+9e )

Followup: MachineOwner

kd> !analyze -v
****************************************************************
*
*
* Bugcheck Analysis
*
*
*
****************************************************************

KERNEL_APC_PENDING_DURING_EXIT (20)
Arguments:
Arg1: 8a210cac, The address of the APC found pending during exit.
Arg2: fffffffe, The thread's APC disable count
Arg3: 00000000, The current IRQL
Arg4: 00000000

Debugging Details:

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0x20

LAST_CONTROL_TRANSFER: from 805f9e4c to 8053331e

STACK_TEXT:
aed7ac50 805f9e4c 00000020 8a210cac fffffffe nt!KeBugCheckEx+0x1b
aed7ad08 8057a46a 00000000 8a14bbc0 00000000
nt!PsWaitForAllProcesses+0x9e
aed7ad28 80582d13 8a14bbc0 00000000 aed7ad64
nt!PspTerminateThreadByPointer+0x52
aed7ad54 804de7ec 00000000 00000000 00eaff24 nt!NtTerminateProcess+0x118

aed7ad54 7c90eb94 00000000 00000000 00eaff24 nt!KiFastCallEntry+0xf8
WARNING: Frame IP not in any known module. Following frames may be
wrong.
00eafe24 00000000 00000000 00000000 00000000 0x7c90eb94

FOLLOWUP_IP:
nt!PsWaitForAllProcesses+9e
805f9e4c 43 inc ebx

SYMBOL_STACK_INDEX: 1

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: nt!PsWaitForAllProcesses+9e

MODULE_NAME: nt

IMAGE_NAME: ntoskrnl.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 42250ff9

STACK_COMMAND: kb

BUCKET_ID: 0x20_nt!PsWaitForAllProcesses+9e

Followup: MachineOwner

kd> kb
ChildEBP RetAddr Args to Child
aed7ac50 805f9e4c 00000020 8a210cac fffffffe nt!KeBugCheckEx+0x1b
aed7ad08 8057a46a 00000000 8a14bbc0 00000000
nt!PsWaitForAllProcesses+0x9e
aed7ad28 80582d13 8a14bbc0 00000000 aed7ad64
nt!PspTerminateThreadByPointer+0x52
aed7ad54 804de7ec 00000000 00000000 00eaff24 nt!NtTerminateProcess+0x118

aed7ad54 7c90eb94 00000000 00000000 00eaff24 nt!KiFastCallEntry+0xf8
WARNING: Frame IP not in any known module. Following frames may be
wrong.
00eafe24 00000000 00000000 00000000 00000000 0x7c90eb94

Questions? First check the IFS FAQ at

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

3

> I have a filter driver to encrypt/decrypt on the fly. One of the

requirement is when each time application reads the encrypted file,
the filter driver need to force it not read from cache, read from disk
instead, so that driver can decrypt data.

I don’t think you will be able to enforce such a requirement. Yes, the cache
will always contain the cleartext data, but why is it bad? The cache is
protected using NT’s ACL-based security mechanisms.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

4

Thank you very much Tony!

I’ll follow your suggestion and add __try __except
structure to driver. Hopefully, I can find some more
information on that testing machine.

Thank you again.

— Tony Mason wrote:

> Do you have structured exception handling set up in
> this driver? This
> error (which is frustrating to track down) could
> occur (for example) if
> outside the scope of the function for which you give
> a code fragment you
> have something like:
>
> try {
>
> // call function implementing the code you provided
>
> } except (EXCEPTION_EXECUTE_HANDLER) {
>
> // handle exception, ignore, convert to error code.
> }
>
> Because my guess is that your Cc/Mm routines are
> raising an error (after
> all, you can’t do what you are trying to do here
> 100% of the time, and
> it sounds like you’ve found one of those
> configurations that leads us to
> tell people “you can’t do it this way”. But we can
> never seem to
> convince anyone of this fact…)
>
> If this is the case I’d suggest that you try putting
> a try/ except
> block around each of your Cc and Mm calls until you
> see which one is
> raising an exception.
>
> Regards,
>
> Tony
>
> Tony Mason
> Consulting Partner
> OSR Open Systems Resources, Inc.
> http://www.osr.com
>

5

Because we have this requirement: only specific
application can read the encrypted file. For example,
using notpad open an encrypted txt file. After that,
if wordpad try to open the same encrypted file, filter
need to enforce wordpad read from disk not from cache.
In that case, filter driver will not decrypt the file.

Thanks

— “Maxim S. Shatskih”
wrote:

> > I have a filter driver to encrypt/decrypt on the
> fly. One of the
> > requirement is when each time application reads
> the encrypted file,
> > the filter driver need to force it not read from
> cache, read from disk
> > instead, so that driver can decrypt data.
>
> I don’t think you will be able to enforce such a
> requirement. Yes, the cache
> will always contain the cleartext data, but why is
> it bad? The cache is
> protected using NT’s ACL-based security mechanisms.
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
>
> —
> Questions? First check the IFS FAQ at
> https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as:
> xxxxx@sbcglobal.net
> To unsubscribe send a blank email to
> xxxxx@lists.osr.com
>

6

Your solution will not work for each case, because if someone map view of
the file then calling
CcPurgeCacheSection will not take effect, because there is no transformation
from the prototype page table entry (PPTE ) to every page table entry(PTE)
which desribes the mapped
page for a process, therefore OS can’t purge the page. This is NT feature,
because some over OS have the ability to do such transformation( e.g.
FreeBSD >5.2 ).

“David Wu” wrote in message news:xxxxx@ntfsd…
> Hi,
> I bring a cache purge issue again even though it has been discussed
> many times.
>
> I have a filter driver to encrypt/decrypt on the fly. One of the
> requirement is when each time application reads the encrypted file,
> the filter driver need to force it not read from cache, read from disk
> instead, so that driver can decrypt data. I use following code someone
> suggested in this forum to purge cache in post IRP_MJ_CREATE (completion
> routine) and force system to launch paging IO to read from disk.
>
> pFCBHeader = (PFSRTL_COMMON_FCB_HEADER)(pIrpStack->FileObject->FsContext);
> pSection = pIrpStack->FileObject->SectionObjectPointer;
>
> KeEnterCriticalRegion();
> ExAcquireResourceExclusiveLite(pFcbHeader->Resource, TRUE);
> ExAcquireResourceExclusiveLite(pFcbHeader->PagingIoResource, TRUE);
>
> CcFlushCache(pSection, 0, 0, NULL);
>
> If (pSection->ImageSectionObject != NULL)
> MmFlushImageSection(pSection, MmFlushForWrite);
>
> If (pSection->DataSectionObject != NULL)
> CcPurgeCacheSection(pSection, NULL, 0, TRUE);
>
> CcUninitializeCacheMap(pIrpStack->FileObject, NULL, NULL);
>
> ExReleaseResourceLite(pFcbHeader->PagingIoResource);
> ExReleaseResourceLite(pFcbHeader->Resource);
> KeLeaveCriticalRegion();
>
> After the cache is purged, I can see IRP_MJ_CLEANUP, IRP_MJ_CLOSE
> and IRP_MJ_READ for paging request. So, it seems cache is purged
> successfully and filter driver can decrypt the file for paging IO.
>
> However, on one of our XP SP2 NTFS testing machine(there are more
> than 20 testing machines set to different configurations), it
> cause KERNEL_APC_PENDING_DURING_EXIT on every time machine reboot
> or shutdown. All other testing machines work fine.
>
> I put the memory dump in here:
> In memory dump, the arg2 is negative. I can not figure out how I call
> KeEnterCriticalRegion() cause APC disable count is not zero. If I
> disable all Cc and Mm functions and just leave resource acquire and
> release in there, the problem won’t happen. Of course, when application
> second time open the encrypted file, it will read from cache, not from
> disk because driver don’t see paging IO. So, it seems that ntoskrnal.exe
> is broken by I call Cc Mm functions. But I can’t get reasonable
> interpretation and don’t know how to solve that.
>
> Do you guys have any ideal? Any suggestion will be appreciated.
> I’m sorry that is long post because I try to give some detail about what I
> did and where the problem is.
>
>
>
> BugCheck 20, {8a210cac, fffffffe, 0, 0}
>
> Probably caused by : ntoskrnl.exe ( nt!PsWaitForAllProcesses+9e )
>
> Followup: MachineOwner
> ---------
>
> kd> !analyze -v
> *****
>
>
> * Bugcheck Analysis
>
>
>
>
>
> KERNEL_APC_PENDING_DURING_EXIT (20)
> Arguments:
> Arg1: 8a210cac, The address of the APC found pending during exit.
> Arg2: fffffffe, The thread’s APC disable count
> Arg3: 00000000, The current IRQL
> Arg4: 00000000
>
> Debugging Details:
> ------------------
>
> DEFAULT_BUCKET_ID: DRIVER_FAULT
>
> BUGCHECK_STR: 0x20
>
> LAST_CONTROL_TRANSFER: from 805f9e4c to 8053331e
>
> STACK_TEXT:
> aed7ac50 805f9e4c 00000020 8a210cac fffffffe nt!KeBugCheckEx+0x1b
> aed7ad08 8057a46a 00000000 8a14bbc0 00000000 nt!PsWaitForAllProcesses+0x9e
> aed7ad28 80582d13 8a14bbc0 00000000 aed7ad64
> nt!PspTerminateThreadByPointer+0x52
> aed7ad54 804de7ec 00000000 00000000 00eaff24 nt!NtTerminateProcess+0x118
> aed7ad54 7c90eb94 00000000 00000000 00eaff24 nt!KiFastCallEntry+0xf8
> WARNING: Frame IP not in any known module. Following frames may be wrong.
> 00eafe24 00000000 00000000 00000000 00000000 0x7c90eb94
>
> FOLLOWUP_IP:
> nt!PsWaitForAllProcesses+9e
> 805f9e4c 43 inc ebx
>
> SYMBOL_STACK_INDEX: 1
>
> FOLLOWUP_NAME: MachineOwner
>
> SYMBOL_NAME: nt!PsWaitForAllProcesses+9e
>
> MODULE_NAME: nt
>
> IMAGE_NAME: ntoskrnl.exe
>
> DEBUG_FLR_IMAGE_TIMESTAMP: 42250ff9
>
> STACK_COMMAND: kb
>
> BUCKET_ID: 0x20_nt!PsWaitForAllProcesses+9e
>
> Followup: MachineOwner
> ---------
>
> kd> kb
> ChildEBP RetAddr Args to Child
> aed7ac50 805f9e4c 00000020 8a210cac fffffffe nt!KeBugCheckEx+0x1b
> aed7ad08 8057a46a 00000000 8a14bbc0 00000000 nt!PsWaitForAllProcesses+0x9e
> aed7ad28 80582d13 8a14bbc0 00000000 aed7ad64
> nt!PspTerminateThreadByPointer+0x52
> aed7ad54 804de7ec 00000000 00000000 00eaff24 nt!NtTerminateProcess+0x118
> aed7ad54 7c90eb94 00000000 00000000 00eaff24 nt!KiFastCallEntry+0xf8
> WARNING: Frame IP not in any known module. Following frames may be wrong.
> 00eafe24 00000000 00000000 00000000 00000000 0x7c90eb94
>
>
>
>

7

I know cache purge will not work for each case. We
have very limited specific applications. It seems that
cache purge works fine on those specific applications
except one case fail. So I’m trying to figure out why
this happen.

Thanks

— Slava Imameyev wrote:

> Your solution will not work for each case, because
> if someone map view of
> the file then calling
> CcPurgeCacheSection will not take effect, because
> there is no transformation
> from the prototype page table entry (PPTE ) to every
> page table entry(PTE)
> which desribes the mapped
> page for a process, therefore OS can’t purge the
> page. This is NT feature,
> because some over OS have the ability to do such
> transformation( e.g.
> FreeBSD >5.2 ).
>
> “David Wu” wrote in
> message news:xxxxx@ntfsd…
> > Hi,
> > I bring a cache purge issue again even though it
> has been discussed
> > many times.
> >
> > I have a filter driver to encrypt/decrypt on the
> fly. One of the
> > requirement is when each time application reads
> the encrypted file,
> > the filter driver need to force it not read from
> cache, read from disk
> > instead, so that driver can decrypt data. I use
> following code someone
> > suggested in this forum to purge cache in post
> IRP_MJ_CREATE (completion
> > routine) and force system to launch paging IO to
> read from disk.
> >
> > pFCBHeader =
>
(PFSRTL_COMMON_FCB_HEADER)(pIrpStack->FileObject->FsContext);
> > pSection =
> pIrpStack->FileObject->SectionObjectPointer;
> >
> > KeEnterCriticalRegion();
> >
> ExAcquireResourceExclusiveLite(pFcbHeader->Resource,
> TRUE);
> >
>
ExAcquireResourceExclusiveLite(pFcbHeader->PagingIoResource,
> TRUE);
> >
> > CcFlushCache(pSection, 0, 0, NULL);
> >
> > If (pSection->ImageSectionObject != NULL)
> > MmFlushImageSection(pSection,
> MmFlushForWrite);
> >
> > If (pSection->DataSectionObject != NULL)
> > CcPurgeCacheSection(pSection, NULL, 0, TRUE);
> >
> > CcUninitializeCacheMap(pIrpStack->FileObject,
> NULL, NULL);
> >
> >
> ExReleaseResourceLite(pFcbHeader->PagingIoResource);
> > ExReleaseResourceLite(pFcbHeader->Resource);
> > KeLeaveCriticalRegion();
> >
> > After the cache is purged, I can see
> IRP_MJ_CLEANUP, IRP_MJ_CLOSE
> > and IRP_MJ_READ for paging request. So, it seems
> cache is purged
> > successfully and filter driver can decrypt the
> file for paging IO.
> >
> > However, on one of our XP SP2 NTFS testing
> machine(there are more
> > than 20 testing machines set to different
> configurations), it
> > cause KERNEL_APC_PENDING_DURING_EXIT on every time
> machine reboot
> > or shutdown. All other testing machines work fine.
> >
> > I put the memory dump in here:
> > In memory dump, the arg2 is negative. I can not
> figure out how I call
> > KeEnterCriticalRegion() cause APC disable count is
> not zero. If I
> > disable all Cc and Mm functions and just leave
> resource acquire and
> > release in there, the problem won’t happen. Of
> course, when application
> > second time open the encrypted file, it will read
> from cache, not from
> > disk because driver don’t see paging IO. So, it
> seems that ntoskrnal.exe
> > is broken by I call Cc Mm functions. But I can’t
> get reasonable
> > interpretation and don’t know how to solve that.
> >
> > Do you guys have any ideal? Any suggestion will be
> appreciated.
> > I’m sorry that is long post because I try to give
> some detail about what I
> > did and where the problem is.
> >
> >
> >
> > BugCheck 20, {8a210cac, fffffffe, 0, 0}
> >
> > Probably caused by : ntoskrnl.exe (
> nt!PsWaitForAllProcesses+9e )
> >
> > Followup: MachineOwner
> > ---------
> >
> > kd> !analyze -v
> >
>
*****
> >
> >
> > * Bugcheck Analysis
> >
> >
> >
> >
>
> >
> > KERNEL_APC_PENDING_DURING_EXIT (20)
> > Arguments:
> > Arg1: 8a210cac, The address of the APC found
> pending during exit.
> > Arg2: fffffffe, The thread’s APC disable count
> > Arg3: 00000000, The current IRQL
> > Arg4: 00000000
> >
> > Debugging Details:
> > ------------------
> >
> > DEFAULT_BUCKET_ID: DRIVER_FAULT
> >
> > BUGCHECK_STR: 0x20
> >
> > LAST_CONTROL_TRANSFER: from 805f9e4c to 8053331e
> >
> > STACK_TEXT:
> > aed7ac50 805f9e4c 00000020 8a210cac fffffffe
> nt!KeBugCheckEx+0x1b
> > aed7ad08 8057a46a 00000000 8a14bbc0 00000000
> nt!PsWaitForAllProcesses+0x9e
> > aed7ad28 80582d13 8a14bbc0 00000000 aed7ad64
> > nt!PspTerminateThreadByPointer+0x52
> > aed7ad54 804de7ec 00000000 00000000 00eaff24
> nt!NtTerminateProcess+0x118
> > aed7ad54 7c90eb94 00000000 00000000 00eaff24
> nt!KiFastCallEntry+0xf8
> > WARNING: Frame IP not in any known module.
> Following frames may be wrong.
> > 00eafe24 00000000 00000000 00000000 00000000
> 0x7c90eb94
> >
> > FOLLOWUP_IP:
> > nt!PsWaitForAllProcesses+9e
> > 805f9e4c 43 inc ebx
> >
> > SYMBOL_STACK_INDEX: 1
> >
> > FOLLOWUP_NAME: MachineOwner
> >
> > SYMBOL_NAME: nt!PsWaitForAllProcesses+9e
> >
> > MODULE_NAME: nt
> >
> > IMAGE_NAME: ntoskrnl.exe
> >
> > DEBUG_FLR_IMAGE_TIMESTAMP: 42250ff9
> >
> > STACK_COMMAND: kb
> >
> > BUCKET_ID: 0x20_nt!PsWaitForAllProcesses+9e
> >
> > Followup: MachineOwner
> > ---------
> >
> > kd> kb
> > ChildEBP RetAddr Args to Child
> > aed7ac50 805f9e4c 00000020 8a210cac fffffffe
> nt!KeBugCheckEx+0x1b
> > aed7ad08 8057a46a 00000000 8a14bbc0 00000000
> nt!PsWaitForAllProcesses+0x9e
> > aed7ad28 80582d13 8a14bbc0 00000000 aed7ad64
> > nt!PspTerminateThreadByPointer+0x52
> > aed7ad54 804de7ec 00000000 00000000 00eaff24
> nt!NtTerminateProcess+0x118
> > aed7ad54 7c90eb94 00000000 00000000 00eaff24
> nt!KiFastCallEntry+0xf8
> > WARNING: Frame IP not in any known module.
> Following frames may be wrong.
> > 00eafe24 00000000 00000000 00000000 00000000
> 0x7c90eb94
> >
> >
> >
> >
>
>
=== message truncated ===

8

I think you should not let data to be read into the cache instead of purging
it every time. Just allocate sector-aligned buffer inside the filter, read
data in it using non-cached IO and then copy data into the application
buffer.

This approach doesn’t work with memory-mapped files, but existence of
memory-mapped file contradicts your requirements, so you should prohibit
memory mapping for protected files anyway.

Alexei.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of David Wu
Sent: Friday, March 10, 2006 7:07 AM
To: Windows File Systems Devs Interest List
Subject: Re: [ntfsd] Cache purge cause KERNEL_APC_PENDING_DURING_EXIT
during shutdown

Because we have this requirement: only specific
application can read the encrypted file. For example,
using notpad open an encrypted txt file. After that,
if wordpad try to open the same encrypted file, filter
need to enforce wordpad read from disk not from cache.
In that case, filter driver will not decrypt the file.

Thanks

— “Maxim S. Shatskih”
wrote:

> > I have a filter driver to encrypt/decrypt on the
> fly. One of the
> > requirement is when each time application reads
> the encrypted file,
> > the filter driver need to force it not read from
> cache, read from disk
> > instead, so that driver can decrypt data.
>
> I don’t think you will be able to enforce such a
> requirement. Yes, the cache
> will always contain the cleartext data, but why is
> it bad? The cache is
> protected using NT’s ACL-based security mechanisms.
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
>
> —
> Questions? First check the IFS FAQ at
> https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as:
> xxxxx@sbcglobal.net
> To unsubscribe send a blank email to
> xxxxx@lists.osr.com
>

—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@vmware.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

9

Alexei,
Your suggestion is very interesting. I’ll consider that while I’m trying to
find out why my driver cause that fault. I just want to make sure I
understand you correctly: Just using buffer in filter to hold non-cached IO
read data. After that, filter decides if copy encrypted or decrypted
data into the application buffer during cache IO.
But here I have one question: sometime, I saw both Irp->MdlAddress and
Irp->UserBuffer are available when IPR_MJ_READ comes in. which buffer the
data should be copied in?
just MdlAddress, UserBuffer or both?

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Alexei Jelvis
Sent: Friday, March 10, 2006 2:28 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Cache purge cause KERNEL_APC_PENDING_DURING_EXIT during
shutdown

I think you should not let data to be read into the cache instead of purging
it every time. Just allocate sector-aligned buffer inside the filter, read
data in it using non-cached IO and then copy data into the application
buffer.

This approach doesn’t work with memory-mapped files, but existence of
memory-mapped file contradicts your requirements, so you should prohibit
memory mapping for protected files anyway.

Alexei.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of David Wu
Sent: Friday, March 10, 2006 7:07 AM
To: Windows File Systems Devs Interest List
Subject: Re: [ntfsd] Cache purge cause KERNEL_APC_PENDING_DURING_EXIT
during shutdown

Because we have this requirement: only specific
application can read the encrypted file. For example,
using notpad open an encrypted txt file. After that,
if wordpad try to open the same encrypted file, filter
need to enforce wordpad read from disk not from cache.
In that case, filter driver will not decrypt the file.

Thanks

— “Maxim S. Shatskih”
wrote:

> > I have a filter driver to encrypt/decrypt on the
> fly. One of the
> > requirement is when each time application reads
> the encrypted file,
> > the filter driver need to force it not read from
> cache, read from disk
> > instead, so that driver can decrypt data.
>
> I don’t think you will be able to enforce such a
> requirement. Yes, the cache
> will always contain the cleartext data, but why is
> it bad? The cache is
> protected using NT’s ACL-based security mechanisms.
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com

10

You should use MdlAddress. UserBuffer is only valid in the context of the
process that originated read. MdlAddress is a kernel space alias for the
UserBuffer.

Alexei.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of David Wu
Sent: Friday, March 10, 2006 5:10 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Cache purge cause KERNEL_APC_PENDING_DURING_EXIT
during shutdown

Alexei,
Your suggestion is very interesting. I’ll consider that while I’m trying to
find out why my driver cause that fault. I just want to make sure I
understand you correctly: Just using buffer in filter to hold non-cached IO
read data. After that, filter decides if copy encrypted or decrypted
data into the application buffer during cache IO.
But here I have one question: sometime, I saw both Irp->MdlAddress and
Irp->UserBuffer are available when IPR_MJ_READ comes in. which buffer the
data should be copied in?
just MdlAddress, UserBuffer or both?

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Alexei Jelvis
Sent: Friday, March 10, 2006 2:28 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Cache purge cause KERNEL_APC_PENDING_DURING_EXIT during
shutdown

I think you should not let data to be read into the cache instead of purging
it every time. Just allocate sector-aligned buffer inside the filter, read
data in it using non-cached IO and then copy data into the application
buffer.

This approach doesn’t work with memory-mapped files, but existence of
memory-mapped file contradicts your requirements, so you should prohibit
memory mapping for protected files anyway.

Alexei.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of David Wu
Sent: Friday, March 10, 2006 7:07 AM
To: Windows File Systems Devs Interest List
Subject: Re: [ntfsd] Cache purge cause KERNEL_APC_PENDING_DURING_EXIT
during shutdown

Because we have this requirement: only specific
application can read the encrypted file. For example,
using notpad open an encrypted txt file. After that,
if wordpad try to open the same encrypted file, filter
need to enforce wordpad read from disk not from cache.
In that case, filter driver will not decrypt the file.

Thanks

— “Maxim S. Shatskih”
wrote:

> > I have a filter driver to encrypt/decrypt on the
> fly. One of the
> > requirement is when each time application reads
> the encrypted file,
> > the filter driver need to force it not read from
> cache, read from disk
> > instead, so that driver can decrypt data.
>
> I don’t think you will be able to enforce such a
> requirement. Yes, the cache
> will always contain the cleartext data, but why is
> it bad? The cache is
> protected using NT’s ACL-based security mechanisms.
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com

—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@vmware.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

11

You can use the following algorithm- if Irp->MdlAddress is NULL then use
Irp->UserBuffer , else use Irp->MdlAddress.
FSDs use both Irp->MdlAddress and Irp->UserBuffer when building a Partial
MDL for a read/write from the disk, they use ( Irp->UserBuffer +
BufferOffset ) as the third parameter in calling IoBuildPartialMdl. The
MDLs from the VMM(i.e. for a Paging IO) may have Mdl->StartVa set to NULL,
therefore Irp->UserBuffer is also NULL.

“David Wu” wrote in message news:xxxxx@ntfsd…
> Alexei,
> Your suggestion is very interesting. I’ll consider that while I’m trying
> to
> find out why my driver cause that fault. I just want to make sure I
> understand you correctly: Just using buffer in filter to hold non-cached
> IO
> read data. After that, filter decides if copy encrypted or decrypted
> data into the application buffer during cache IO.
> But here I have one question: sometime, I saw both Irp->MdlAddress and
> Irp->UserBuffer are available when IPR_MJ_READ comes in. which buffer the
> data should be copied in?
> just MdlAddress, UserBuffer or both?
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Alexei Jelvis
> Sent: Friday, March 10, 2006 2:28 PM
> To: Windows File Systems Devs Interest List
> Subject: RE: [ntfsd] Cache purge cause KERNEL_APC_PENDING_DURING_EXIT
> during
> shutdown
>
> I think you should not let data to be read into the cache instead of
> purging
> it every time. Just allocate sector-aligned buffer inside the filter, read
> data in it using non-cached IO and then copy data into the application
> buffer.
>
> This approach doesn’t work with memory-mapped files, but existence of
> memory-mapped file contradicts your requirements, so you should prohibit
> memory mapping for protected files anyway.
>
> Alexei.
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com]On Behalf Of David Wu
> Sent: Friday, March 10, 2006 7:07 AM
> To: Windows File Systems Devs Interest List
> Subject: Re: [ntfsd] Cache purge cause KERNEL_APC_PENDING_DURING_EXIT
> during shutdown
>
>
> Because we have this requirement: only specific
> application can read the encrypted file. For example,
> using notpad open an encrypted txt file. After that,
> if wordpad try to open the same encrypted file, filter
> need to enforce wordpad read from disk not from cache.
> In that case, filter driver will not decrypt the file.
>
> Thanks
>
> — “Maxim S. Shatskih”
> wrote:
>
>> > I have a filter driver to encrypt/decrypt on the
>> fly. One of the
>> > requirement is when each time application reads
>> the encrypted file,
>> > the filter driver need to force it not read from
>> cache, read from disk
>> > instead, so that driver can decrypt data.
>>
>> I don’t think you will be able to enforce such a
>> requirement. Yes, the cache
>> will always contain the cleartext data, but why is
>> it bad? The cache is
>> protected using NT’s ACL-based security mechanisms.
>>
>> Maxim Shatskih, Windows DDK MVP
>> StorageCraft Corporation
>> xxxxx@storagecraft.com
>> http://www.storagecraft.com
>
>
>

12

I think that FSD’s cached IO IRPs have no MdlAddress at all.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Slava Imameyev”
Newsgroups: ntfsd
To: “Windows File Systems Devs Interest List”
Sent: Saturday, March 11, 2006 10:15 AM
Subject: Re:[ntfsd] Cache purge cause KERNEL_APC_PENDING_DURING_EXIT during
shutdown

> You can use the following algorithm- if Irp->MdlAddress is NULL then use
> Irp->UserBuffer , else use Irp->MdlAddress.
> FSDs use both Irp->MdlAddress and Irp->UserBuffer when building a Partial
> MDL for a read/write from the disk, they use ( Irp->UserBuffer +
> BufferOffset ) as the third parameter in calling IoBuildPartialMdl. The
> MDLs from the VMM(i.e. for a Paging IO) may have Mdl->StartVa set to NULL,
> therefore Irp->UserBuffer is also NULL.
>
>
> “David Wu” wrote in message news:xxxxx@ntfsd…
> > Alexei,
> > Your suggestion is very interesting. I’ll consider that while I’m trying
> > to
> > find out why my driver cause that fault. I just want to make sure I
> > understand you correctly: Just using buffer in filter to hold non-cached
> > IO
> > read data. After that, filter decides if copy encrypted or decrypted
> > data into the application buffer during cache IO.
> > But here I have one question: sometime, I saw both Irp->MdlAddress and
> > Irp->UserBuffer are available when IPR_MJ_READ comes in. which buffer the
> > data should be copied in?
> > just MdlAddress, UserBuffer or both?
> >
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of Alexei Jelvis
> > Sent: Friday, March 10, 2006 2:28 PM
> > To: Windows File Systems Devs Interest List
> > Subject: RE: [ntfsd] Cache purge cause KERNEL_APC_PENDING_DURING_EXIT
> > during
> > shutdown
> >
> > I think you should not let data to be read into the cache instead of
> > purging
> > it every time. Just allocate sector-aligned buffer inside the filter, read
> > data in it using non-cached IO and then copy data into the application
> > buffer.
> >
> > This approach doesn’t work with memory-mapped files, but existence of
> > memory-mapped file contradicts your requirements, so you should prohibit
> > memory mapping for protected files anyway.
> >
> > Alexei.
> >
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com]On Behalf Of David Wu
> > Sent: Friday, March 10, 2006 7:07 AM
> > To: Windows File Systems Devs Interest List
> > Subject: Re: [ntfsd] Cache purge cause KERNEL_APC_PENDING_DURING_EXIT
> > during shutdown
> >
> >
> > Because we have this requirement: only specific
> > application can read the encrypted file. For example,
> > using notpad open an encrypted txt file. After that,
> > if wordpad try to open the same encrypted file, filter
> > need to enforce wordpad read from disk not from cache.
> > In that case, filter driver will not decrypt the file.
> >
> > Thanks
> >
> > — “Maxim S. Shatskih”
> > wrote:
> >
> >> > I have a filter driver to encrypt/decrypt on the
> >> fly. One of the
> >> > requirement is when each time application reads
> >> the encrypted file,
> >> > the filter driver need to force it not read from
> >> cache, read from disk
> >> > instead, so that driver can decrypt data.
> >>
> >> I don’t think you will be able to enforce such a
> >> requirement. Yes, the cache
> >> will always contain the cleartext data, but why is
> >> it bad? The cache is
> >> protected using NT’s ACL-based security mechanisms.
> >>
> >> Maxim Shatskih, Windows DDK MVP
> >> StorageCraft Corporation
> >> xxxxx@storagecraft.com
> >> http://www.storagecraft.com
> >
> >
> >
>
>
>
> —
> Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com

13



In fact many of CcXxx, MmXxx and FsRtlXxx routines raise different
exceptions (but almost in very critical cases except for FsRtlXxx
subset) and this behavior is not always properly described in the
DDK/IFS documentation. So using try/ except blocks in such cases is
a must. And also do not forget to release all previously acquired
resources and leave critical region even if you gracefully processed
some exception (that’s
why you have negative APC disable count in the system dump and this
bugcheck code). Using
try/ finally blocks may help you when try/ except block is not
located in the very function that calls “dangerous” routines. In the
case of possible exception the __finally{} block will be called before
control is transferred to the exception handler and you will be able to
easily release all resources and leave critical region.

David Wu wrote:

Thank you very much Tony!

I’ll follow your suggestion and add try except
structure to driver. Hopefully, I can find some more
information on that testing machine.

Thank you again.

— Tony Masonwrote:

Do you have structured exception handling set up in
this driver? This
error (which is frustrating to track down) could
occur (for example) if
outside the scope of the function for which you give
a code fragment you
have something like:

__try {

// call function implementing the code you provided

} __except (EXCEPTION_EXECUTE_HANDLER) {

// handle exception, ignore, convert to error code.
}

Because my guess is that your Cc/Mm routines are
raising an error (after
all, you can’t do what you are trying to do here
100% of the time, and
it sounds like you’ve found one of those
configurations that leads us to
tell people “you can’t do it this way”. But we can
never seem to
convince anyone of this fact…)

If this is the case I’d suggest that you try putting
a try/ except
block around each of your Cc and Mm calls until you
see which one is
raising an exception.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

14

I would wrap any Cc call and MmProbeAndLockPages in __try/__except and
return the exception code as NTSTATUS. Creating a wrapper functions for this is
also a good idea. I think that using exception failures mode in CcXxx is NT’s
misdesign.

These wrappers should change the exception failure mode to
error-status-return failure mode, which is by far more convinient and readable.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Konstantin Manurin”
Newsgroups: ntfsd
To: “Windows File Systems Devs Interest List”
Sent: Saturday, March 11, 2006 10:02 PM
Subject: Re:[ntfsd] Cache purge cause KERNEL_APC_PENDING_DURING_EXIT during
shutdown

>
> In fact many of CcXxx, MmXxx and FsRtlXxx routines raise different exceptions
(but almost in very critical cases except for FsRtlXxx subset) and this
behavior is not always properly described in the DDK/IFS documentation. So
using try/ except blocks in such cases is a must. And also do not forget to
release all previously acquired resources and leave critical region even if you
gracefully processed some exception (that’s why you have negative APC disable
count in the system dump and this bugcheck code). Using try/ finally blocks
may help you when try/ except block is not located in the very function that
calls “dangerous” routines. In the case of possible exception the finally{}
block will be called before control is transferred to the exception handler and
you will be able to easily release all resources and leave critical region.
>
>
> David Wu wrote:
> Thank you very much Tony!
>
> I’ll follow your suggestion and add try __except
> structure to driver. Hopefully, I can find some more
> information on that testing machine.
>
> Thank you again.
>
> — Tony Mason wrote:
>
> Do you have structured exception handling set up in
> this driver? This
> error (which is frustrating to track down) could
> occur (for example) if
> outside the scope of the function for which you give
> a code fragment you
> have something like:
>
>__try {
>
> // call function implementing the code you provided
>
> } __except (EXCEPTION_EXECUTE_HANDLER) {
>
> // handle exception, ignore, convert to error code.
> }
>
> Because my guess is that your Cc/Mm routines are
> raising an error (after
> all, you can’t do what you are trying to do here
> 100% of the time, and
> it sounds like you’ve found one of those
> configurations that leads us to
> tell people “you can’t do it this way”. But we can
> never seem to
> convince anyone of this fact…)
>
> If this is the case I’d suggest that you try putting
> a__try/__except
> block around each of your Cc and Mm calls until you
> see which one is
> raising an exception.
>
> Regards,
>
> Tony
>
> Tony Mason
> Consulting Partner
> OSR Open Systems Resources, Inc.
> http://www.osr.com
>
>
>
>
>
> —
> Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com

15

Thank you so much for all you guys advice. I’ll try on those suggestions on
next week.

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Konstantin Manurin
Sent: Saturday, March 11, 2006 1:03 PM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Cache purge cause KERNEL_APC_PENDING_DURING_EXIT during
shutdown

In fact many of CcXxx, MmXxx and FsRtlXxx routines raise different
exceptions (but almost in very critical cases except for FsRtlXxx subset)
and this behavior is not always properly described in the DDK/IFS
documentation. So using __try/__except blocks in such cases is a must. And
also do not forget to release all previously acquired resources and leave
critical region even if you gracefully processed some exception (that’s why
you have negative APC disable count in the system dump and this bugcheck
code). Using __try/__finally blocks may help you when __try/__except block
is not located in the very function that calls “dangerous” routines. In the
case of possible exception the __finally{} block will be called before
control is transferred to the exception handler and you will be able to
easily release all resources and leave critical region.

David Wu wrote:

Thank you very much Tony!

I’ll follow your suggestion and add __try __except
structure to driver. Hopefully, I can find some more
information on that testing machine.

Thank you again.

— Tony Mason mailto:xxxxx wrote:

Do you have structured exception handling set up in
this driver? This
error (which is frustrating to track down) could
occur (for example) if
outside the scope of the function for which you give
a code fragment you
have something like:

try {

// call function implementing the code you provided

} except (EXCEPTION_EXECUTE_HANDLER) {

// handle exception, ignore, convert to error code.
}

Because my guess is that your Cc/Mm routines are
raising an error (after
all, you can’t do what you are trying to do here
100% of the time, and
it sounds like you’ve found one of those
configurations that leads us to
tell people “you can’t do it this way”. But we can
never seem to
convince anyone of this fact…)

If this is the case I’d suggest that you try putting
a try/ except
block around each of your Cc and Mm calls until you
see which one is
raising an exception.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@sbcglobal.net
To unsubscribe send a blank email to xxxxx@lists.osr.com</mailto:xxxxx>