Scott Noone pointed out to me the fact that guarded mutexes are used in
2k3 (described here: http://www.osronline.com/article.cfm?article=283).
After thinking about it and discussing it with a couple of folks, I’m
convinced that I don’t understand what the change was for. Can anyone
shed some light on the relative benefits of using guarded mutexes?
Thanks.
-sd
Steve Dispensa
MVP - Windows DDK
www.kernelmustard.com
The big change here is that it eliminates the use of APC_LEVEL for
normal mutexes and fast mutexes but still blocks ALL APCs.
The disadvantage of using IRQL APC_LEVEL is that it required
reprogramming the CPU Task Priority Register (TPR) that is used for
routing interrupts in the I/O APIC on an MP system. By eliminating
this, it improves performance of the system. Thus, the OS folks decided
to introduce a new synchronization mechanism that blocks ALL APCs but
leaves the IRQL at PASSIVE_LEVEL.
There are two reasons this hits us in file systems:
(1) We’ve all learned that you can’t call ZwXxx files to perform I/O at
APC_LEVEL. However, the docs say you can do so at PASSIVE_LEVEL. But
that is no longer true - you can only do it at PASSIVE_LEVEL if special
kernel APCs are not disabled. Of course, currently there is no way to
test and see if special kernel APCs are disabled.
(2) Mm has changed much of its code to use guarded mutexes rather than
fast mutexes. Thus, paging I/O operations no longer arrive at APC_LEVEL
but rather at PASSIVE_LEVEL.
We have seen several deadlocks that arise as a result of this in drivers
that have code to detect APC_LEVEL and defend against it do not have
code to detect the blocking of special kernel APCs.
Currently, the only call is KeAreApcsDisabled and this returns TRUE even
if only kernel APCs are disabled
(KeEnterCriticalRegion/FsRtlEnterFileSystem has been called) but not
special kernel APCs.
I hope this helps.
Regards,
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com
Looking forward to seeing you at the Next OSR File Systems Class October
18, 2004 in Silicon Valley!
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Steve Dispensa
Sent: Thursday, September 30, 2004 7:38 AM
To: ntfsd redirect
Subject: [ntfsd] Guarded Mutexes
Scott Noone pointed out to me the fact that guarded mutexes are used in
2k3 (described here: http://www.osronline.com/article.cfm?article=283).
After thinking about it and discussing it with a couple of folks, I’m
convinced that I don’t understand what the change was for. Can anyone
shed some light on the relative benefits of using guarded mutexes?
Thanks.
-sd
Steve Dispensa
MVP - Windows DDK
www.kernelmustard.com
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
On Sep 30, 2004, at 6:47 AM, Tony Mason wrote:
The disadvantage of using IRQL APC_LEVEL is that it required
reprogramming the CPU Task Priority Register (TPR) that is used for
routing interrupts in the I/O APIC on an MP system. By eliminating
this, it improves performance of the system. Thus, the OS folks
decided
to introduce a new synchronization mechanism that blocks ALL APCs but
leaves the IRQL at PASSIVE_LEVEL.
So what’s left that *does* raise the IRQL to APC_LEVEL? I always
thought of “blocks all APCs” as the precise purpose of APC_LEVEL.
Currently, the only call is KeAreApcsDisabled and this returns TRUE
even
if only kernel APCs are disabled
(KeEnterCriticalRegion/FsRtlEnterFileSystem has been called) but not
special kernel APCs.
If this is the case, the article
(http://www.osronline.com/article.cfm?article=283) is either misleading
or wrong; I got the impression KeAreApcsDisabled() tested for special
kernel APCs from it.
on amd64:
lkd> u keareapcsdisabled
nt!KeAreApcsDisabled:
fffff80000826780 65488b042588010000 mov rax,gs:[00000188] fffff80000826789 83b8bc00000000 cmp dword ptr [rax+0xbc],0x0
fffff80000826790 0f95c0 setne al fffff80000826793 c3 ret
lkd> dt _KTHREAD
…
+0x0bc KernelApcDisable : Int2B
+0x0be SpecialApcDisable : Int2B
+0x0bc CombinedApcDisable : Uint4B
Guessing:
union {
struct {
USHORT KernelApcDisable;
USHORT SpecialApcDisable;
}
ULONG CombinedApcDisable;
}
Looks to me like it tests both USHORTs there by doing a dword compare
on CombinedApcDisable.
I hope this helps.
Always.
-sd
Steve,
My point is only that KeAreApcsDisabled returns TRUE if kernel APCs are
disabled but special kernel APCs are not. There’s never a case where
special kernel APCs are disabled but regular kernel APCs are not. In
Windows 2003 there is no call to determine if SPECIAL kernel APCs are
disabled using this new mechanism.
Regards,
Tony
Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com
Looking forward to seeing you at the Next OSR File Systems Class October
18, 2004 in Silicon Valley!
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Steve Dispensa
Sent: Thursday, September 30, 2004 8:11 AM
To: ntfsd redirect
Subject: Re: [ntfsd] Guarded Mutexes
On Sep 30, 2004, at 6:47 AM, Tony Mason wrote:
The disadvantage of using IRQL APC_LEVEL is that it required
reprogramming the CPU Task Priority Register (TPR) that is used for
routing interrupts in the I/O APIC on an MP system. By eliminating
this, it improves performance of the system. Thus, the OS folks
decided
to introduce a new synchronization mechanism that blocks ALL APCs but
leaves the IRQL at PASSIVE_LEVEL.
So what’s left that *does* raise the IRQL to APC_LEVEL? I always
thought of “blocks all APCs” as the precise purpose of APC_LEVEL.
Currently, the only call is KeAreApcsDisabled and this returns TRUE
even
if only kernel APCs are disabled
(KeEnterCriticalRegion/FsRtlEnterFileSystem has been called) but not
special kernel APCs.
If this is the case, the article
(http://www.osronline.com/article.cfm?article=283) is either misleading
or wrong; I got the impression KeAreApcsDisabled() tested for special
kernel APCs from it.
on amd64:
lkd> u keareapcsdisabled
nt!KeAreApcsDisabled:
fffff80000826780 65488b042588010000 mov rax,gs:[00000188] fffff80000826789 83b8bc00000000 cmp dword ptr [rax+0xbc],0x0
fffff80000826790 0f95c0 setne al fffff80000826793 c3 ret
lkd> dt _KTHREAD
…
+0x0bc KernelApcDisable : Int2B
+0x0be SpecialApcDisable : Int2B
+0x0bc CombinedApcDisable : Uint4B
Guessing:
union {
struct {
USHORT KernelApcDisable;
USHORT SpecialApcDisable;
}
ULONG CombinedApcDisable;
}
Looks to me like it tests both USHORTs there by doing a dword compare
on CombinedApcDisable.
I hope this helps.
Always.
-sd
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
On Sep 30, 2004, at 7:35 AM, Tony Mason wrote:
Steve,
My point is only that KeAreApcsDisabled returns TRUE if kernel APCs are
disabled but special kernel APCs are not. There’s never a case where
special kernel APCs are disabled but regular kernel APCs are not. In
Windows 2003 there is no call to determine if SPECIAL kernel APCs are
disabled using this new mechanism.
Good point 
coffee++;
-sd
This is a good example of what happens when the kernel guys make changes
to the system without talking to the file system guys about it.
I new about this issue but had not read the OSR article yet. Thanks for
posting the link to it.
Srv03 SP1 will have a new API called KeAreAllApcsDisabled(). This API
will return TRUE if IRQL level is APC or higher or if special kernel
APCs are disabled. I know this doesn’t help you a lot now but the
problem will be fixed.
I wish to make one correction to the OSR article. It implied to me that
you should start calling KeAreApcsDisabled INSTEAD of querying the
current IRQL. Until you can use the new API available in Srv03 SP1 you
actually need to do check both the IRQL level AND KeAreApcsDisabled().
Since MM is the only component using this you only need to do this
double check when dealing with Paging IO.
Steve asked the question “So what’s left that *does* raise the IRQL to
APC_LEVEL?” The answer for now is everything but MM.
Going forward more and more components will be using guarded mutexes due
to the perf advantage. For example in longhorn NTFS replaced all usages
of fast mutexes with guarded mutexes.
Now is the time to start adapting your filters to the changes that are
coming.
FYI, I have filed a documentation bug for Srv03 SP1 to get the docs
updated on this issue.
Neal Christiansen
Microsoft File System Filter Group Lead
This posting is provided “AS IS” with no warranties, and confers no
rights
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Steve Dispensa
Sent: Thursday, September 30, 2004 6:18 AM
To: Windows File Systems Devs Interest List
Subject: Re: [ntfsd] Guarded Mutexes
On Sep 30, 2004, at 7:35 AM, Tony Mason wrote:
Steve,
My point is only that KeAreApcsDisabled returns TRUE if kernel APCs
are
disabled but special kernel APCs are not. There’s never a case where
special kernel APCs are disabled but regular kernel APCs are not. In
Windows 2003 there is no call to determine if SPECIAL kernel APCs are
disabled using this new mechanism.
Good point
coffee++;
-sd
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
> The disadvantage of using IRQL APC_LEVEL is that it required
reprogramming the CPU Task Priority Register (TPR) that is used for
routing interrupts in the I/O APIC on an MP system. By eliminating
It is very strange that APC and DISPATCH IRQLs are implemented via hardware.
Looks like nothing prevents the HAL authors to implement them in software only,
and touch the APIC only when the IRQL transition is > DISPATCH.
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com