Hello all,
I’ve encountered what appears at first glance to be a bug in the Windows XP kernel. I’m trying to understand why it’s happening but I don’t have a ton of kernel experience so I was wondering if anyone had some tips.
Ultimately the problem that is occurring is that I am using an application (not written by me and for which I don’t have source) that somehow or another causes my entire OS to hang. I attached a kernel debugger and figured out that when the hang occurs, I always have a very specific call stack that looks like the following:
nt!MmRaisePoolQuota
nt!PspExpandQuota
nt!PsChargeSharedPoolQuota
…
kernel32!CreateFileMapping
However PsChargeSharedPoolQuota contains an infinite loop, so when MmRaisePoolQuota returns 1,it causes PspExpandQuota to return 1, and then PsChargeSharedPoolQuota just goes back to the top of the loop and repeats. So basically the kernel never returns from PsChargeSharedPoolQuota and eats 100% of my CPU. Every call to PspExpandQuota always receives exactly the same arguments, and every call to MmRaisePoolQuota in turn always receives the exact arguments.
All 3 of these functions are undocumented, and while I’ve already almost completely reversed all 3 of them, I’m still having trouble understanding why the problem is occuring, partly because they make use of some private kernel structures that I don’t understand.
Was wondering if anyone could shed some light. In particular, the following structures come into play:
struct EPROCESS_QUOTA_BLOCK
{
EPROCESS_QUOTA_ENTRY QuotaEntry[3];
LIST_ENTRY QuotaList;
ULONG ReferenceCount;
ULONG ProcessCount;
};
struct EPROCESS_QUOTA_ENTRY
{
ULONG Usage;
ULONG Limit;
ULONG Peak;
ULONG Return;
};
Some of these fields are self explanatory (Usage, Limit, Peak), but some are not (Return, ReferenceCount, ProcessCount). Also the functions in question always access block.QuotaEntry[1], and always ignore QuotaEntry[0] and QuotaEntry[2]. I don’t understand why this is either.
Anyone have any thoughts and/or experience, with these 2 kernel structures, or the 3 functions? From what I can deduce the signatures of these functions are:
DWORD PsChargeSharedPoolQuota(_EPROCESS* proc, int paged_amt, int nonpaged_amt);
DWORD PspExpandQuota(BOOL is_paged, EPROCESS_QUOTA_ENTRY* entry, int usage, int charge, ???)
DWORD MmRaisePoolQuota(BOOL is_paged, int limit, int ???)
ftr the app that’s causing this to happen is Visual Studio 2010 beta
