IRQL rules for RtlXxxString functions

The documentation for RtlXxxString and friends in most cases says IRQL must be PASSIVE_LEVEL. Is this a hard and fast rule that the verifier/PREfast/checked build is going to trip me up on, or is some flexibility allowed?

If I can guarantee that the buffers for my UNICODE_STRING are NonPagedPool, can I get away with calling RtlCompareUnicodeString? Or is it more that the RtlCompareUnicodeString routine itself might be paged out?

My DeviceIdentification consists of two UNICODE_STRINGs of arbitrary length, so they need to be managed separately to the DeviceIdentification. The names reflect the name of a network resource that in turn has an arbitrary length (there probably is an actual limit, but it’s probably longer than I want to allocate “just in case”).

Thanks

James

I wouldn’t toy with RtlXXXString IRQL requirements if you can find another
way.

In some situations it may be possible to upcase the strings at some prior
point at passive level (or when they are defined). If you can pull this off,
then you can 1.) check the Length and then 2.) if Length matches use
RtlCompareMemory on the Buffer at DISPATCH_LEVEL.

Thomas F. Divine

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of James Harper
Sent: Saturday, December 7, 2013 11:46 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] IRQL rules for RtlXxxString functions

The documentation for RtlXxxString and friends in most cases says IRQL must
be PASSIVE_LEVEL. Is this a hard and fast rule that the
verifier/PREfast/checked build is going to trip me up on, or is some
flexibility allowed?

If I can guarantee that the buffers for my UNICODE_STRING are NonPagedPool,
can I get away with calling RtlCompareUnicodeString? Or is it more that the
RtlCompareUnicodeString routine itself might be paged out?

My DeviceIdentification consists of two UNICODE_STRINGs of arbitrary length,
so they need to be managed separately to the DeviceIdentification. The names
reflect the name of a network resource that in turn has an arbitrary length
(there probably is an actual limit, but it’s probably longer than I want to
allocate “just in case”).

Thanks

James


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

>If I can guarantee that the buffers for my UNICODE_STRING are NonPagedPool,

can I get away with >calling RtlCompareUnicodeString? Or is it more that
the RtlCompareUnicodeString routine itself might >be paged out?

The routine may be paged out but also the character tables are said to
reside in paged memory. If you don’t need this for sorting but use this only
to check if two strings are equal, instead of using RtlCompareUnicodeString,
consider using RtlCompareMemory on the non paged buffers instead.

//Daniel

> If I can guarantee that the buffers for my UNICODE_STRING are NonPagedPool, can I get away

with calling RtlCompareUnicodeString? Or is it more that the RtlCompareUnicodeString routine itself
might be paged out?

Let’s speculate a bit. IIRC, PASSIVE_LEVEL requirement means that you are not allowed to invoke any code subjected to this constraint on the paging IO path to the pagefile (which, IIRC, runs at APC_LEVEL) . This, in turn, implies that the code in question either may be pageable itself, or it may access pagefile-backed data behind the scenes. Otherwise, IRQL requirement to accessing pageable memory should be <=DISPATCH_LEVEL due solely to inability to block at elevated IRQL…

To summarize, calling a function with PASSIVE_LEVEL constraint at any other IRQL seems to be pretty unwise…

Anton Bassov

The tables required for sorting are huge, and I would expect them to be
pageable. There is no need to use the CompareString if equality is all
that is required, bearing in mind that some characters that have aliased
codes in Unicode, such as “.”, might compare as “equal” under the “compare
Unicode string” function and “not equal” under the “compare memory”
option. Note that the “Code Red” exploit used this technique to bypass
the checking for “…” in the cgi path, thus allowing the exploit to escape
confinement. The “…” it checked for was by looking for two of the Page
00 “.” characters (U002E, if I remember my codes right, which might not be
the case…), so the exploit used a dot character from another Unicode
page, which was not seen as “…” by the checking software, but WAS seen as
“…” by the rest of the file system, hence the ability to break out of the
sandbox.
joe

>If I can guarantee that the buffers for my UNICODE_STRING are
> NonPagedPool,
>can I get away with >calling RtlCompareUnicodeString? Or is it more that
>the RtlCompareUnicodeString routine itself might >be paged out?

The routine may be paged out but also the character tables are said to
reside in paged memory. If you don’t need this for sorting but use this
only
to check if two strings are equal, instead of using
RtlCompareUnicodeString,
consider using RtlCompareMemory on the non paged buffers instead.

//Daniel


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

I think the Unicode tables are pageable, and thus this limitation.

You can compare the strings bytewise (case-sensitive) at any IRQL using memcmp() or such.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

“James Harper” wrote in message news:xxxxx@ntdev…
The documentation for RtlXxxString and friends in most cases says IRQL must be PASSIVE_LEVEL. Is this a hard and fast rule that the verifier/PREfast/checked build is going to trip me up on, or is some flexibility allowed?

If I can guarantee that the buffers for my UNICODE_STRING are NonPagedPool, can I get away with calling RtlCompareUnicodeString? Or is it more that the RtlCompareUnicodeString routine itself might be paged out?

My DeviceIdentification consists of two UNICODE_STRINGs of arbitrary length, so they need to be managed separately to the DeviceIdentification. The names reflect the name of a network resource that in turn has an arbitrary length (there probably is an actual limit, but it’s probably longer than I want to allocate “just in case”).

Thanks

James

>

I think the Unicode tables are pageable, and thus this limitation.

That would explain the requirement.

You can compare the strings bytewise (case-sensitive) at any IRQL using
memcmp() or such.

I’m using the name of the storage resource as the device identification. The storage resource name is ascii, so I’m having the usermode pass it to the kernel as ascii so it is always converted to the same unicode (I sanitise the ascii first), so a RtlCompareMemory should be fine.

Except that converting to Unicode isn’t something I really wanted to do so I was hoping for usermode to do the conversion and pass the Unicode into the kernel directly. Because the conversion would be done in the context of the usermode process locale though, two processes could have the same ascii that results in different Unicode, I think. Is there a way I can reliably sanitise the Unicode without requiring any of the RtlXxx string routines?

Thanks

James

Note that case-insensitive compare is extremely complex in Unicode, so if
it is required you have to obey the PASSIVE_LEVEL constraint.

Otherwise, if bitwise equality suffices, a bitwise compare will do just fine.
joe

I think the Unicode tables are pageable, and thus this limitation.

You can compare the strings bytewise (case-sensitive) at any IRQL
using memcmp() or such.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

“James Harper” wrote in message
> news:xxxxx@ntdev…
> The documentation for RtlXxxString and friends in most cases says IRQL
> must be PASSIVE_LEVEL. Is this a hard and fast rule that the
> verifier/PREfast/checked build is going to trip me up on, or is some
> flexibility allowed?
>
> If I can guarantee that the buffers for my UNICODE_STRING are
> NonPagedPool, can I get away with calling RtlCompareUnicodeString? Or is
> it more that the RtlCompareUnicodeString routine itself might be paged
> out?
>
> My DeviceIdentification consists of two UNICODE_STRINGs of arbitrary
> length, so they need to be managed separately to the DeviceIdentification.
> The names reflect the name of a network resource that in turn has an
> arbitrary length (there probably is an actual limit, but it’s probably
> longer than I want to allocate “just in case”).
>
> Thanks
>
> James
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

> Because the conversion would be done in the context of the usermode process locale though,

two processes could have the same ascii that results in different Unicode, I think.

Actually, I have always thought locale works on per-system basis, rather than per process one.
Am I wrong?

Anton Bassov

>

> Because the conversion would be done in the context of the usermode
> process locale though,
> two processes could have the same ascii that results in different Unicode, I
> think.

Actually, I have always thought locale works on per-system basis, rather than
per process one.
Am I wrong?

I meant that if different user processes did the conversion then it could result in different Unicode representations.

James

> I meant that if different user processes did the conversion then it could result in different

Unicode representations.

…but my question still stands - I have always thought locale concept works on SYSTEM-WIDE basis, so that you cannot have locale A for the user X and locale Y for the user B. Things like user preferences, home directory, etc are user-specific, but locale is not (otherwise Logon screen may be displayed in language that a given user does not understand) Is this suggestion correct?

Anton Bassov

> Actually, I have always thought locale works on per-system basis, rather than per process one.

The “Language for Unicode programs” settings, which governs MBCS<->Unicode conversions, is system-wide.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

On 09-Dec-2013 17:12, Maxim S. Shatskih wrote:

> Actually, I have always thought locale works on per-system basis, rather than per process one.

The “Language for Unicode programs” settings, which governs MBCS<->Unicode conversions, is system-wide.

http://www.microsoft.com/en-us/download/details.aspx?id=13209

  • pa

Locale is per thread:
http://msdn.microsoft.com/en-us/library/windows/desktop/dd374051(v=vs.85).aspx

– pa

On 09-Dec-2013 14:49, xxxxx@hotmail.com wrote:

> I meant that if different user processes did the conversion then it could result in different
> Unicode representations.

…but my question still stands - I have always thought locale concept works on SYSTEM-WIDE basis, so that you cannot have locale A for the user X and locale Y for the user B. Things like user preferences, home directory, etc are user-specific, but locale is not (otherwise Logon screen may be displayed in language that a given user does not understand) Is this suggestion correct?

Anton Bassov

I also have some questions about safe strings. Could someone clarify if these functions can be called at DISPATCH?

I saw a few threads (AFAIR with replies from Doron) saying that if format string doesn’t contain any string specifiers (like %s and so on) the functions can be called at DISPATCH. On the other hand I saw a thread describing BSoD even with no string specifiers. Probably the code path is PAGED.

What a bigger problem I am trying to solve?

Debug prints are not enough sometimes. I would like to have some logging with ETW mechanism to know what happened before OS has died. So, it should be something like KdPrint but not only to debug port but also to ETW. RtlStringCchVPrintf looks very useful to format the string, but some functions may be called at DISPATCH, so it should be DISPATCH safe.

I could re-implement the wheel with my own DISPATCH safe formatter function. But it would be better to use existing one.

Thanks.

I believe it is per-thread, with each thread starting off with the locale
of the process, which is based on the per-session locale, which defaults
to the system locale (the API SetThreadLocale is kind of a giveaway on
this…)
joe

> Because the conversion would be done in the context of the usermode
> process locale though,
> two processes could have the same ascii that results in different
> Unicode, I think.

Actually, I have always thought locale works on per-system basis, rather
than per process one.
Am I wrong?

Anton Bassov


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Per-thread. See SetThreadLocale. Also look at WideCharToMultiByte and
MultiByteToWideChar, which can supply a locale, indicate the current
thread locale, current session locale, or current system locale be used.
joe

On 09-Dec-2013 17:12, Maxim S. Shatskih wrote:
>> Actually, I have always thought locale works on per-system basis,
>> rather than per process one.
>
> The “Language for Unicode programs” settings, which governs
> MBCS<->Unicode conversions, is system-wide.

http://www.microsoft.com/en-us/download/details.aspx?id=13209

  • pa

NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

I think that the change of “Language for non-Unicode programs” in Control Panel actually installs another NLS files to the OS.

So, I expect that the only locales which will work with W2A and A2W functions are US English and the one installed in the Control Panel. Well, also UTF-8 which is considered to be a fake locale, just to reuse W2A and A2W to do Unicode <-> UTF-8 conversions.

IE supports displaying lots of different national languages in the web pages, but I don’t think it ever supports A encoding for these languages, I think it only works with W and UTF-8.

So, if you need to work with some national language using some app, and the app is built using A encoding, then you must set the aforementioned Control Panel setting.

Am I wrong?


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

wrote in message news:xxxxx@ntdev…
> Per-thread. See SetThreadLocale. Also look at WideCharToMultiByte and
> MultiByteToWideChar, which can supply a locale, indicate the current
> thread locale, current session locale, or current system locale be used.
> joe
>
>> On 09-Dec-2013 17:12, Maxim S. Shatskih wrote:
>>>> Actually, I have always thought locale works on per-system basis,
>>>> rather than per process one.
>>>
>>> The “Language for Unicode programs” settings, which governs
>>> MBCS<->Unicode conversions, is system-wide.
>>
>> http://www.microsoft.com/en-us/download/details.aspx?id=13209
>>
>> - pa
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>>
>> OSR is HIRING!! See http://www.osr.com/careers
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>
>
>

> I think that the change of “Language for non-Unicode programs” in

Control Panel actually installs another NLS files to the OS.

So, I expect that the only locales which will work with W2A and A2W
functions are US English and the one installed in the Control Panel.
Well, also UTF-8 which is considered to be a fake locale, just to
reuse W2A and A2W to do Unicode <-> UTF-8 conversions.

I believe A2W and W2A use the current user’s locale, not the system
locale, so they default to the per-session. But I haven’t looked at the
expansion of those macros in many years.

IE supports displaying lots of different national languages in the web
pages, but I don’t think it ever supports A encoding for these
languages, I think it only works with W and UTF-8.

So, if you need to work with some national language using some app,
and the app is built using A encoding, then you must set the
aforementioned Control Panel setting.

Yep. This is why 8-bit apps should be treated as another dangerous and
dead technology.

Or, as I would tell my students, “The correct response, when your boss
comes in and says ‘We have a massive order for our software from Korea,
but they want it localized. How long will that take?’ the correct answer
is not ‘Well, let me see if I can get it to compile in Unicode, and then
see how many things break, and I’ll get back to you, give me a month or
so’ but ‘How long will it take to get the Korean translator to get here?’”

While Unicode is still a pretty small piece of the localization effort, if
it a critical piece, and without it, there is not much point to worrying
about currency formats, date formats, time formats, numeric formats in
general, etc.
joe

Am I wrong?


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

wrote in message news:xxxxx@ntdev…
>> Per-thread. See SetThreadLocale. Also look at WideCharToMultiByte and
>> MultiByteToWideChar, which can supply a locale, indicate the current
>> thread locale, current session locale, or current system locale be used.
>> joe
>>
>>> On 09-Dec-2013 17:12, Maxim S. Shatskih wrote:
>>>>> Actually, I have always thought locale works on per-system basis,
>>>>> rather than per process one.
>>>>
>>>> The “Language for Unicode programs” settings, which governs
>>>> MBCS<->Unicode conversions, is system-wide.
>>>
>>> http://www.microsoft.com/en-us/download/details.aspx?id=13209
>>>
>>> - pa
>>>
>>>
>>> —
>>> NTDEV is sponsored by OSR
>>>
>>> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>>>
>>> OSR is HIRING!! See http://www.osr.com/careers
>>>
>>> For our schedule of WDF, WDM, debugging and other seminars visit:
>>> http://www.osr.com/seminars
>>>
>>> To unsubscribe, visit the List Server section of OSR Online at
>>> http://www.osronline.com/page.cfm?name=ListServer
>>>
>>
>>
>>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

> I believe A2W and W2A use the current user’s locale, not the system

locale, so they default to the per-session. But I haven’t looked at the
expansion of those macros in many years.

I just nicknamed the MultiByteXxx functions this way, to avoid excessive typing :slight_smile:

but they want it localized. How long will that take?’ the correct answer
is not ‘Well, let me see if I can get it to compile in Unicode, and then
see how many things break, and I’ll get back to you, give me a month or
so’ but ‘How long will it take to get the Korean translator to get here?’"

Modern days, compiling apps to ANSI is a nonsense.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com