Debugging Win32K.sys

NTDEV
1

Greetings,

Now I realise that I’m asking a question about Win32 rather than device
driver development - but I’m utterly stumped.

I have an application that calls

HDESK hDesk = CreateDesktop(“MyDesktop”, NULL, NULL, 0,
DESKTOP_ALL_ACCESS, NULL);

to create a desktop. On some machines, this fails with
ERROR_NOT_ENOUGH_MEMORY. The machine I have at the moment where it fails is
W2K3 with Terminal Services installed.

Having stepped through the disassembler of CreateDesktop() I find the
following:

7FFE0300 8B D4 mov edx,esp
7FFE0302 0F 34 sysenter
7FFE0304 C3 ret

with eax holding 0x1153

Since this is a switch to Ring0, I thought that you guys might be able to
help with any ideas about how to step through the kernel side of
CreateDesktop.

Obviously, if any of you have any clues as to why CreateDesktop is returning
ERROR_NOT_ENOUGH_MEMORY, that would be great!

Thanks!
Richie Oak

2

Richard Oak wrote:

Greetings,

Now I realise that I’m asking a question about Win32 rather than
device driver development - but I’m utterly stumped.

I have an application that calls

HDESK hDesk = CreateDesktop(“MyDesktop”, NULL, NULL, 0,
DESKTOP_ALL_ACCESS, NULL);

to create a desktop. On some machines, this fails with
ERROR_NOT_ENOUGH_MEMORY. The machine I have at the moment where it
fails is W2K3 with Terminal Services installed.

…

Obviously, if any of you have any clues as to why CreateDesktop is
returning ERROR_NOT_ENOUGH_MEMORY, that would be great!

Hmmm, there’s a thread from someone asking this question in 2003, but no
response.

The MSDN page on CreateDesktop describes the memory limitations and how
to hack the registry to get around them. That would be my first stop:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/createdesktop.asp

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.