Placement new operator in a driver

NTDEV
41

NICK:

My apologies about the confusion, probably on my part. I wasn’t
attempting to comment on the compilers actions; in fact, I agree 100%
both that (1) there is no standard governing and (2) accordingly, it is
either (1) correct or (2) N/A, depending on how you look at it.

I thought you meant that the compiler could put the autogenerated code
anywhere it wished, without it affecting you. This, depending on what
you do and where you do it, is not true. That is all I was trying to
say.

It sounds like we are all on the same page; this is probably not a very
good idea for most circumstances.

>> xxxxx@cristalink.com 2006-04-23 20:10 >>>

“Martin O’Brien” wrote in message
news:xxxxx@ntdev…
> For example, does inability to
> mix exception handling types refer to the immediate caller, or the
> entire call chain (including library functions).

I don’t use exception handlers, apart from those for
MmProbeAndLockPages or
similar. I don’t think it’s something very useful for kernel mode.

>> The compiler is free to put an implicitly generated function to
wherever
>> it wants.

> This is, in my opinion, just incorrect. You will not feel that way
if

To my knowledge, the relation between autogenerated code and the
segment it
is placed in is not documented or defined in any standard (please
correct me
if I am wrong here). Consequently, the compiler can put it anywere.You
may
not like it (btw, I hate it), but this does not make that behavior
‘incorrect’, of course, if ‘incorrect’ is an opposite to ‘correct’, and
the
latter is ‘conforming to an approved or conventional standard’
(Merriam-Webster’s definition).

>
> This is, in my opinion, just incorrect. You will not feel that way
if
> an implicitly defined destructor in a base class, such as a vector
> deletion, appears in the wrong order and causes code to end up in a
> pageable section. This is, in my opinion, not at all easy to
predict,
> and very difficult to diagnose, particuarly if you use some else code
in
> your hierarchy without knowing whether the destructor is virtual or
> not.
>
> MM
>
>

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

42

First, I would use new(&Lookaside), not new(Memory) - lesser code to type.

Second - write

void* operator new(size_t, PNPAGED_LOOKASIDE_LIST Lookaside)
{
return ExAllocateFromNPagedLookasideList(Lookaside);
}

Third - I would do this function a class method of the particular class.

Fourth - are you sure C++ will bail out properly if “operator new” will
return NULL, and will not proceed constructing the object? If it will proceed -
then sorry, this is a BSOD, and you should not use “operator new” in the
drivers.

Usually, an exception is thrown if “operator new” fails, but sorry, no C++
exceptions in the kernel.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Edouard A.”
To: “Windows System Software Devs Interest List”
Sent: Saturday, April 22, 2006 11:21 AM
Subject: [ntdev] Placement new operator in a driver

> Hi all,
>
> My question for today is for people who do C++ in drivers. In my project I
> developed an object oriented linked list (amongst other things) to replace
> the existing linked list present in the DDK (I have my reasons, main one
> being nevrosis). It works. However I came accross a memory optimization
> problem, when I add an object my template does something like :
>
> NTSTATUS AddEntry(T **ppAddedEntry, T *pToAdd)
> {
>
> T *pNewEntry = new T(pToAdd)
>
> // linked list chaining stuff and about 700 lines of ASSERT()
>
> *ppAddedEntry = pNewEntry;
>
> return STATUS_SUCCESS;
>
> }
>
> Don’t worry, my driver has got specific new and delete operator that will
> take the memory from paged pool or non paged pool (depending on the object).
>
> I told to myself : “wait, won’t it be more efficient to use a lookaside list
> to allocate memory for all the entries of the list since they have the same
> size and might be very small omg this sentence is long”. So I did that. But
> if I allocate the memory this way, the constructor of the object will not be
> called which is of course a problem.
>
> “No problem!” says I “I’ll use a placement new”, then I did that :
>
> NTSTATUS AddEntry(…)
> {
>
> void *pMemory = static_cast> *>(ExAllocateLookAsideListOptimizationOmgOmg(…))
>
> // …
>
> T *pNewEntry = new(pMemory)T();
>
> }
>
> And now the compiler doesn’t like me anymore and tells me awful things like
> “there is no new which takes two arguments” and “I was faking”.
>
> Is there no support for placement new operator in the DDK or is it me who
> missed something? It works 100% with the former method but I thought look
> aside lists could have a positive effect on memory usage.
>
> Thanks for your help.
>
> –
>
> Edouard
>
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

43

> Curious, but what is the purpose of C++ in a driver? Other than added

problems such as this, why?

This holy war is too long, and the guy really needed help.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

44

>(*) I was just debugging another component as a favor to another team

and it took me *two* days of looking over the code to see that there was
an overloaded operator=() that was doing all the magic. Not the best
use of my time nor the best use of the language.

Doron, “operator T” is much worse.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

45

> I think your being extravagant. This could all be done with a

plugboard.

No Martin, most drivers would require at least several plugboards for the
equivalent functionality. Conditional logic was not one of the strong
points of plugboard programming.

Loren

(And yes, the first programming I did was using plugboards for about 20
different machines.)

46

My apologies. Just kidding. Well before my time.

>> xxxxx@earthlink.net 2006-04-23 21:02 >>>
I think your being extravagant. This could all be done with a
plugboard.

No Martin, most drivers would require at least several plugboards for
the
equivalent functionality. Conditional logic was not one of the strong
points of plugboard programming.

Loren

(And yes, the first programming I did was using plugboards for about
20
different machines.)

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

47

I would agree, overloaded casting operators (T*, T&, const T&) are
beyond evil. They are like the little black rock at the end of Time
Bandits…concentrated evil ;).

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S.
Shatskih
Sent: Sunday, April 23, 2006 5:49 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Placement new operator in a driver

(*) I was just debugging another component as a favor to another team
and it took me *two* days of looking over the code to see that there
was
an overloaded operator=() that was doing all the magic. Not the best
use of my time nor the best use of the language.

Doron, “operator T” is much worse.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

48

I would agree that passing the lookaside to operator new is better for 2
reasons. First, it makes it obvious what is being used to allocate
memory. Second, prefast/SDV think that operator new() can fail b/c it
is marked as a memory allocator. By passing the lookaside, you still
check for a NULL return and it makes the tools happy. If you just pass
valid memory to new() and return it (which means it can’t fail) and you
don’t check for NULL, the tools complain (and no, there is no SAL to say
that it can’t fail so be quiet).

Operator new() will only throw exceptions if you enable it in the
compiler. It is opt in (At least for KM, UM might be reversed).

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S.
Shatskih
Sent: Sunday, April 23, 2006 5:29 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Placement new operator in a driver

First, I would use new(&Lookaside), not new(Memory) - lesser code to
type.

Second - write

void* operator new(size_t, PNPAGED_LOOKASIDE_LIST Lookaside)
{
return ExAllocateFromNPagedLookasideList(Lookaside);
}

Third - I would do this function a class method of the particular
class.

Fourth - are you sure C++ will bail out properly if “operator new”
will
return NULL, and will not proceed constructing the object? If it will
proceed -
then sorry, this is a BSOD, and you should not use “operator new” in the
drivers.

Usually, an exception is thrown if “operator new” fails, but sorry,
no C++
exceptions in the kernel.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Edouard A.”
To: “Windows System Software Devs Interest List”
Sent: Saturday, April 22, 2006 11:21 AM
Subject: [ntdev] Placement new operator in a driver

> Hi all,
>
> My question for today is for people who do C++ in drivers. In my
project I
> developed an object oriented linked list (amongst other things) to
replace
> the existing linked list present in the DDK (I have my reasons, main
one
> being nevrosis). It works. However I came accross a memory
optimization
> problem, when I add an object my template does something like :
>
> NTSTATUS AddEntry(T **ppAddedEntry, T *pToAdd)
> {
>
> T *pNewEntry = new T(pToAdd)
>
> // linked list chaining stuff and about 700 lines of ASSERT()
>
> *ppAddedEntry = pNewEntry;
>
> return STATUS_SUCCESS;
>
> }
>
> Don’t worry, my driver has got specific new and delete operator that
will
> take the memory from paged pool or non paged pool (depending on the
object).
>
> I told to myself : “wait, won’t it be more efficient to use a
lookaside list
> to allocate memory for all the entries of the list since they have the
same
> size and might be very small omg this sentence is long”. So I did
that. But
> if I allocate the memory this way, the constructor of the object will
not be
> called which is of course a problem.
>
> “No problem!” says I “I’ll use a placement new”, then I did that :
>
> NTSTATUS AddEntry(…)
> {
>
> void *pMemory = static_cast> *>(ExAllocateLookAsideListOptimizationOmgOmg(…))
>
> // …
>
> T *pNewEntry = new(pMemory)T();
>
> }
>
> And now the compiler doesn’t like me anymore and tells me awful things
like
> “there is no new which takes two arguments” and “I was faking”.
>
> Is there no support for placement new operator in the DDK or is it me
who
> missed something? It works 100% with the former method but I thought
look
> aside lists could have a positive effect on memory usage.
>
> Thanks for your help.
>
> –
>
> Edouard
>
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

49

Autogenerated code lives all over the place, some you can avoid, some
you can’t. It includes scalar and vector deletes, both of which you
cannot write on your own. If the linker puts these in the wrong
section, you are hosed.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of cristalink
Sent: Sunday, April 23, 2006 4:05 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Re:Placement new operator in a driver

According to my experience, all code, including templates, goes to a
default
segment, which is non-paged. If one starts to mess with autogenerated
code,
such as constructors, destructors, copy constructors etc, they are
looking
for troubles. There is not much point in having constructors or
destructors
paged, as they are usually short, at least my ones.

the documentation for DriverWorks, if you have access to it

No, thank you :slight_smile:

–

“Martin O’Brien” wrote in message
news:xxxxx@ntdev…
>I haven’t looked at this problem in quite a while, as my experiences
> with them convinced me to avoid them where I was not willing to check
> out the generated code extensively. I can give you an example for
> virtual destructors; not for templates (I explain why later). The
basic
> problem with virtual destructors is the same as that with templates:
> code can end up in pageable segments, which can (and generally will)
> cause problems that I’m sure your familiar with. The significant
> difference is that the manifestations of the case of virtual
destructors
> can be, to a significant extent, predicted and worked around. That
> being said, I rarely do so, as it is cumbersome, ugly, quite possibly
> likely to break when the compiler changes, and, in the case of the
> kernel, avoidable by changing the representation. Normally, this is,
in
> my opinion, about the worst thing one can do; however, in this case,
the
> reasons are cogent. Unfortunately, as my principal motivation for
using
> C++ in the kernel (anywhere I roam, really) is templates, this is of
no
> help here.
>
> Please keep in mind that this is all empirical, and likely to change
> with the compiler.
>
> For every version of VC++ that I have looked at closely (which does
not
> include the most recent one, and, possibly the penultimate one as
well),
> the section in to which the code for a destructor is emitted depends
on
> whether it is virtual or not. In the case of a non virtual
destructor,
> the code (appears, at least) to go in to the same section as the first
> code that references it; for the virtual case, it (again, appears) to
go
> in to the section that contains a reference to a constructor. The
> specific situation in which this causes a problem is when the first
> constructor (for a class with a virtual destructor) appears in the
INIT
> (or other discardable) section, and reference to the virtual
destructor
> occurs after it is discarded. The particularly nasty (and not
> automatably addresable) complication of this is that this will occur
if
> any of the classes base classes (if any) satisfy the same conditions
and
> appear in the INIT section. There is no reasonable way to ensure that
> this does not occur, other than to avoid the use of virtual
destructors
> completely, or using the following, rather brutal hack, that, as I
> mentioned, I do not use (with a few specific exceptions that I have
> researched extensively; that being said, it is probably still not the
> best idea, but rather one that I have gotten away with, and very
> possibly will be able to continue to do so, at least in the very
> meaningful sense (in my particular case, which is unusual, but,
> unfortunately, I may not describe)). Additional fun can be had
because
> the compiler will generate destructors for scalar and vector deletion,
> and these, if unspecified and not worked around, will end up in INIT
> under the same conditions. OK, one more twist. The later only occurs
> for checked builds.
>
> A work around to this, such as it is, is that to do the best you can
to
> ensure this issue, for any class that uses a virtual destructor, and
the
> at least one constructor for this class will occur in a pageable
section
> (under the conditions described above), you must create another
> constructor (that may just be empty) that appears in the same source
> module as the one that will appear in the INIT section, and it must
> occur before that constructor. As mentioned above, but reiterated
here
> because it is very easy to overlook, this is true for any class that
> derives from a class which contains a virtual destructor. If all of
> this sounds very suspect, I would tell you that it seems to work, and
> then agree with you. The only confirmation, such as it is, that I can
> offer you, is that I came up with this independently of DriverWorks
(but
> not, by a long shot, on my own), and we seem to agree. Normally, this
> would be comforting; however, as has been pointed out many times in
this
> list (and I don’t think I have ever seen anyone disagree), DriverWorks
> is a proverbial Piece of Shit, so I don’t know what this really
offers.
>
> I’m sorry, but I can not offer you a concrete example that I know
> exhibits it, because of the nature of the work in which I am involved.
> This, however, is, to the best of my understanding and research,
> entirely accurate, including the part about it being labile.
>
> The situation for templates is much, much worse, in the sense of
being,
> without access to the source code for the compiler in question, there
is
> really no way (that I know of) to detail this conditions that dictate
> where any piece of instantiated templated code might appear.
Formally,
> the whole issue is a Halting problem, and, as many such issues due,
> almost assuredly does have an answer, but not one that can be relied
> upon, unless, of course, you happen to know it. Ironically, and some
> would, quite reasonably, say foolishly, I basically don’t use virtual
> destructors, and I do, quite causiously, use templates.
>
> The long and short of this is that, to be safe as possible, you have
to
> look at complete map file, searching for the particulars, taking in
to
> account the base class issue. This is very difficult, as well as time
> consuming, so I have to really, really want something to do it. I
have
> written a tool (or modified an existing tool) that does, reliably, it
> seems, process images using the symbols to verify that particular
> instances of this problem are not occuring. This, of course, only
works
> by my specifying accurately and without fail, what it should look for.
>
> I don’t imagine this helped very much, but this is, unfortunately, the
> best I have to offer. The best source of information I can point you
to
> on this, is the documentation for DriverWorks, if you have access to
it.
> I, presumably, may not include or note it, due to licensing issues.
> And, finally, as noted above, the product, well, sucks. The long and
> short is that, using these features of C++ (not to mention exception
> handling), needs to be carefully weighed against the potential
downside.
> In my case, which is, I think, unusual, I find it to be worth it.
>
> MM
>

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

50

As martin said, FORCEINLINE does not work b/c on chk builds it is a
no-op. Furthermore, let’s say GetCount() was this

KIRQL irql;
ULONG count;

LockObject(&irql); // acquires object spinlock, this code is inlined
count = m_Count;
UnlockObjecT(irql); // releases spinlock

return count;

now you are inlining spinlock code. If you call GetCount() and it is
inlined from a pageable function, you are hosed again.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of cristalink
Sent: Sunday, April 23, 2006 3:32 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Placement new operator in a driver

>The problem with templates is that the #pragmas around them do not
apply

To my experience, templates are always placed in a non paged segment. I
would like to see sample code that proves otherwise. Besides, this may
depend on the compiler version. What was a problem in VC6 may no longer
apply in VC7.

>ULONG GetCount(VOID) { return m_Count; }

The solution is to use __forceinline, the “inline only those marked as
inline” switch, and to make sure the relevant warnings are enabled
(“function selected for inline expansion”, “function not inlined” etc).

>destructors

To my experience, constructors and destructors should (a) always be
explicitly defined, and (b) defined non-paged in .cpp. The compiler is
free
to put an implicitly generated function to wherever it wants.

Cheers

–

“Doron Holan” wrote in message
news:xxxxx@ntdev…
The problem with templates is that the #pragmas around them do not
apply. Where the code is instantiated is a non deterministic. I worked
on a failed project that used templates in a driver where the section in
which the generated code for a template moved from build to build
depending on minor changes elsewhere in the binary. It would blow up at
random spots where code was not supposed to be pageable, but it was.
Basically, the only workaround is to have a completely non pageable
driver.

As for virtual destructors, the gotcha here is that the linker remove
identical functions. Many classes virtual destructor is the exactly the
same code so you see only one or two of them that are shared in the
entire class hierarchy. The pageability of this function is not a
criteria for uniqueness so you can get a virtual desctructor that should
be NP put in a pageable section. Templates make this worse b/c of the
even more randomness of the section that they generate code in.

Note that this does not apply only the desctructors but any function.
Another likely candidate are accessor functions like this

ULONG GetCount(VOID) { return m_Count; }

Where m_Count’s offset is 0x4. Any other accessor function which
returns a ULONG where the field resides at offset 0x4 will share the
function implementation even though they are not in the same class.
This also has the nasty side affect of making callstacks look very
strange and can lead to some initial false positives, such as

Bar::~Bar()
Foo::DestroySelf()
Foo::RunDown()

Where ~Bar() is a virtual destructor and where ~Foo() matches it, so the
compiler just uses ~Bar() instead of ~Foo().

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of cristalink
Sent: Sunday, April 23, 2006 1:45 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Placement new operator in a driver

Can you be more specific, please? Can you provide any example of a
templated
function placed in a paged segment? What was the problem with virtual
destructors?

Thanks

–

“Martin O’Brien” wrote in message
news:xxxxx@ntdev…
> This is definitely true. I learned about this one and the related
> problem of virtual destructors in the hardest of possible ways. That
> being said, while there are very real constraints on what you can
safely
> do, and you really, really must either understand them or being
willing
> to suffer until you do, I still use aspects of C++ in the kernel,
mostly
> because I like to.
>
>
>
>>>> xxxxx@microsoft.com 2006-04-22 13:13 >>>
> You can get 1) and 2) just by using a .cpp file and then using C BTW.
> /W4 also gets you most of the way in C.
>
>> However I came accross a memory optimization problem, when I add an
> object my template does something like :
>
> Please tell me you are not using templates. If you are don’t try to
> make any function pageable, otherwise the linker may place templacized
> code in the wrong segment.
>
> d
>

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

51

  1. accessor: no, it is any function in any file that matches the
    assembly signature. That is cross hierarchy, cross object, it could
    even be a non object function if it matches.

  2. inlining is also ND and if you are inlining functions which acquire
    spinlocks in pageable code, you can get hosed quickly.

  3. yes

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Martin O’Brien
Sent: Sunday, April 23, 2006 3:20 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Placement new operator in a driver

DORON:

I just wrote a long, fairly unclear, explanation to the dev on my
experience with this subject. You, as, in addition to describing the
same issues I mentioned (both at greater at length yet considerably less
clearly than you), have identified some very troubling ones that I have
never even considered, seem to know quite a bit more about this than I.
I was wondering, for my own benefit, if not the dev’s, if you wouldn’t
mind reading what I wrote and telling me what you think. Even though,
based on stability of the work I do (which is entirely research), I
don’t think I’ve run in to the accessor issue, I would greatly
appreciate this, as it sounds like, effectively, a deal breaker, the
tool I’ve written and the fact that I make use of the template features
(and no others that I am aware of) only when it just irritates too much
to go without. Not the best reason, but what can I tell you. In
particular, I’m confused about a few things.

  1. For the accessor issue, this is true only for functions within the
    heirarchy?
  2. How does inlining (aside from unpredictably) factor in to this?
  3. Purely as a hypothetical (as I can imagine ever doing this), could
    a pair of non member functions that act essentially as accessors to
    global variable(s) exhibit this behavior? If forced to come up an
    example, I suppose that accessors to some sort of global union could do
    this, if it applies.

This is not exactly pressing stuff, but, if you wouldn’t mind, I would
really appreciate you input.

Thanks,

M. M. O’Brien

>> xxxxx@microsoft.com 2006-04-23 17:22 >>>
The problem with templates is that the #pragmas around them do not
apply. Where the code is instantiated is a non deterministic. I
worked
on a failed project that used templates in a driver where the section
in
which the generated code for a template moved from build to build
depending on minor changes elsewhere in the binary. It would blow up
at
random spots where code was not supposed to be pageable, but it was.
Basically, the only workaround is to have a completely non pageable
driver.

As for virtual destructors, the gotcha here is that the linker remove
identical functions. Many classes virtual destructor is the exactly
the
same code so you see only one or two of them that are shared in the
entire class hierarchy. The pageability of this function is not a
criteria for uniqueness so you can get a virtual desctructor that
should
be NP put in a pageable section. Templates make this worse b/c of the
even more randomness of the section that they generate code in.

Note that this does not apply only the desctructors but any function.
Another likely candidate are accessor functions like this

ULONG GetCount(VOID) { return m_Count; }

Where m_Count’s offset is 0x4. Any other accessor function which
returns a ULONG where the field resides at offset 0x4 will share the
function implementation even though they are not in the same class.
This also has the nasty side affect of making callstacks look very
strange and can lead to some initial false positives, such as

Bar::~Bar()
Foo::DestroySelf()
Foo::RunDown()

Where ~Bar() is a virtual destructor and where ~Foo() matches it, so
the
compiler just uses ~Bar() instead of ~Foo().

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of cristalink
Sent: Sunday, April 23, 2006 1:45 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Placement new operator in a driver

Can you be more specific, please? Can you provide any example of a
templated
function placed in a paged segment? What was the problem with virtual
destructors?

Thanks

–

“Martin O’Brien” wrote in message
news:xxxxx@ntdev…
> This is definitely true. I learned about this one and the related
> problem of virtual destructors in the hardest of possible ways.
That
> being said, while there are very real constraints on what you can
safely
> do, and you really, really must either understand them or being
willing
> to suffer until you do, I still use aspects of C++ in the kernel,
mostly
> because I like to.
>
>
>
>>>> xxxxx@microsoft.com 2006-04-22 13:13 >>>
> You can get 1) and 2) just by using a .cpp file and then using C
BTW.
> /W4 also gets you most of the way in C.
>
>> However I came accross a memory optimization problem, when I add an
> object my template does something like :
>
> Please tell me you are not using templates. If you are don’t try to
> make any function pageable, otherwise the linker may place
templacized
> code in the wrong segment.
>
> d
>

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

52

Structured Exception Handling (SEH) is also considered exception
handling in addition to C++ EH so anywhere it is present, inlining will
not occur.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of cristalink
Sent: Sunday, April 23, 2006 5:10 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Re:Placement new operator in a driver

“Martin O’Brien” wrote in message
news:xxxxx@ntdev…
> For example, does inability to
> mix exception handling types refer to the immediate caller, or the
> entire call chain (including library functions).

I don’t use exception handlers, apart from those for MmProbeAndLockPages
or
similar. I don’t think it’s something very useful for kernel mode.

>> The compiler is free to put an implicitly generated function to
wherever
>> it wants.

> This is, in my opinion, just incorrect. You will not feel that way if

To my knowledge, the relation between autogenerated code and the segment
it
is placed in is not documented or defined in any standard (please
correct me
if I am wrong here). Consequently, the compiler can put it anywere.You
may
not like it (btw, I hate it), but this does not make that behavior
‘incorrect’, of course, if ‘incorrect’ is an opposite to ‘correct’, and
the
latter is ‘conforming to an approved or conventional standard’
(Merriam-Webster’s definition).

>
> This is, in my opinion, just incorrect. You will not feel that way if
> an implicitly defined destructor in a base class, such as a vector
> deletion, appears in the wrong order and causes code to end up in a
> pageable section. This is, in my opinion, not at all easy to predict,
> and very difficult to diagnose, particuarly if you use some else code
in
> your hierarchy without knowing whether the destructor is virtual or
> not.
>
> MM
>
>

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

53

Thanks, Doron.

MM

>> xxxxx@microsoft.com 2006-04-23 22:37 >>>

  1. accessor: no, it is any function in any file that matches the
    assembly signature. That is cross hierarchy, cross object, it could
    even be a non object function if it matches.

  2. inlining is also ND and if you are inlining functions which
    acquire
    spinlocks in pageable code, you can get hosed quickly.

  3. yes

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Martin O’Brien
Sent: Sunday, April 23, 2006 3:20 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Placement new operator in a driver

DORON:

I just wrote a long, fairly unclear, explanation to the dev on my
experience with this subject. You, as, in addition to describing the
same issues I mentioned (both at greater at length yet considerably
less
clearly than you), have identified some very troubling ones that I
have
never even considered, seem to know quite a bit more about this than I.

I was wondering, for my own benefit, if not the dev’s, if you wouldn’t
mind reading what I wrote and telling me what you think. Even though,
based on stability of the work I do (which is entirely research), I
don’t think I’ve run in to the accessor issue, I would greatly
appreciate this, as it sounds like, effectively, a deal breaker, the
tool I’ve written and the fact that I make use of the template
features
(and no others that I am aware of) only when it just irritates too
much
to go without. Not the best reason, but what can I tell you. In
particular, I’m confused about a few things.

  1. For the accessor issue, this is true only for functions within
    the
    heirarchy?
  2. How does inlining (aside from unpredictably) factor in to this?
  3. Purely as a hypothetical (as I can imagine ever doing this), could
    a pair of non member functions that act essentially as accessors to
    global variable(s) exhibit this behavior? If forced to come up an
    example, I suppose that accessors to some sort of global union could
    do
    this, if it applies.

This is not exactly pressing stuff, but, if you wouldn’t mind, I would
really appreciate you input.

Thanks,

M. M. O’Brien

>> xxxxx@microsoft.com 2006-04-23 17:22 >>>
The problem with templates is that the #pragmas around them do not
apply. Where the code is instantiated is a non deterministic. I
worked
on a failed project that used templates in a driver where the section
in
which the generated code for a template moved from build to build
depending on minor changes elsewhere in the binary. It would blow up
at
random spots where code was not supposed to be pageable, but it was.
Basically, the only workaround is to have a completely non pageable
driver.

As for virtual destructors, the gotcha here is that the linker remove
identical functions. Many classes virtual destructor is the exactly
the
same code so you see only one or two of them that are shared in the
entire class hierarchy. The pageability of this function is not a
criteria for uniqueness so you can get a virtual desctructor that
should
be NP put in a pageable section. Templates make this worse b/c of the
even more randomness of the section that they generate code in.

Note that this does not apply only the desctructors but any function.
Another likely candidate are accessor functions like this

ULONG GetCount(VOID) { return m_Count; }

Where m_Count’s offset is 0x4. Any other accessor function which
returns a ULONG where the field resides at offset 0x4 will share the
function implementation even though they are not in the same class.
This also has the nasty side affect of making callstacks look very
strange and can lead to some initial false positives, such as

Bar::~Bar()
Foo::DestroySelf()
Foo::RunDown()

Where ~Bar() is a virtual destructor and where ~Foo() matches it, so
the
compiler just uses ~Bar() instead of ~Foo().

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of cristalink
Sent: Sunday, April 23, 2006 1:45 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Placement new operator in a driver

Can you be more specific, please? Can you provide any example of a
templated
function placed in a paged segment? What was the problem with virtual
destructors?

Thanks

–

“Martin O’Brien” wrote in message
news:xxxxx@ntdev…
> This is definitely true. I learned about this one and the related
> problem of virtual destructors in the hardest of possible ways.
That
> being said, while there are very real constraints on what you can
safely
> do, and you really, really must either understand them or being
willing
> to suffer until you do, I still use aspects of C++ in the kernel,
mostly
> because I like to.
>
>
>
>>>> xxxxx@microsoft.com 2006-04-22 13:13 >>>
> You can get 1) and 2) just by using a .cpp file and then using C
BTW.
> /W4 also gets you most of the way in C.
>
>> However I came accross a memory optimization problem, when I add an
> object my template does something like :
>
> Please tell me you are not using templates. If you are don’t try to
> make any function pageable, otherwise the linker may place
templacized
> code in the wrong segment.
>
> d
>

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

54

> Autogenerated code lives all over the place, some you can avoid, some

you can’t. It includes scalar and vector deletes
Both C and C++ were born as essentially sequential languages, and kernel
is very multithreaded (if one can be “more” or “less” multithreaded…).

Neither C nor C++ knows irql.

I’d agree with Jan that occasionally C++ may make things a little
easier,
and btw use the same kind of refcounted template with due prayers,
but I was thinking, given that parallel execution becomes a buzz word,
is there a chance that a language will be updated?

Java and .NET got a syntax for locks, as opposed to a game we play
with lock-in-ctor, unlock-in-dtor.

I understand this is impractical, if at all possible, but different
semantics
simply begs for a new language, at least with built-in notion of levels
of interruptability, with built-in notion of pageability and with syntax
elements
reflecting thread-related semantics. Not in my lifetime, of course.

Anyway, the strangest fact about this whole discussion is that it actually
brought up nontrivial things. Normally this type of argument - “C [C++]
is better” is, how can I say, much less productive.

OSR, how about summing it up in an article in the “insider”?
IMHO, Jan’s and Doron’s posts plus the rest that was said deserve it.

----- Original Message -----
From: “Doron Holan”
To: “Windows System Software Devs Interest List”
Sent: Sunday, April 23, 2006 10:34 PM
Subject: RE: [ntdev] Re:Placement new operator in a driver

Autogenerated code lives all over the place, some you can avoid, some
you can’t. It includes scalar and vector deletes, both of which you
cannot write on your own. If the linker puts these in the wrong
section, you are hosed.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of cristalink
Sent: Sunday, April 23, 2006 4:05 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Re:Placement new operator in a driver

According to my experience, all code, including templates, goes to a
default
segment, which is non-paged. If one starts to mess with autogenerated
code,
such as constructors, destructors, copy constructors etc, they are
looking
for troubles. There is not much point in having constructors or
destructors
paged, as they are usually short, at least my ones.

>the documentation for DriverWorks, if you have access to it

No, thank you :slight_smile:

–

“Martin O’Brien” wrote in message
news:xxxxx@ntdev…
>I haven’t looked at this problem in quite a while, as my experiences
> with them convinced me to avoid them where I was not willing to check
> out the generated code extensively. I can give you an example for
> virtual destructors; not for templates (I explain why later). The
basic
> problem with virtual destructors is the same as that with templates:
> code can end up in pageable segments, which can (and generally will)
> cause problems that I’m sure your familiar with. The significant
> difference is that the manifestations of the case of virtual
destructors
> can be, to a significant extent, predicted and worked around. That
> being said, I rarely do so, as it is cumbersome, ugly, quite possibly
> likely to break when the compiler changes, and, in the case of the
> kernel, avoidable by changing the representation. Normally, this is,
in
> my opinion, about the worst thing one can do; however, in this case,
the
> reasons are cogent. Unfortunately, as my principal motivation for
using
> C++ in the kernel (anywhere I roam, really) is templates, this is of
no
> help here.
>
> Please keep in mind that this is all empirical, and likely to change
> with the compiler.
>
> For every version of VC++ that I have looked at closely (which does
not
> include the most recent one, and, possibly the penultimate one as
well),
> the section in to which the code for a destructor is emitted depends
on
> whether it is virtual or not. In the case of a non virtual
destructor,
> the code (appears, at least) to go in to the same section as the first
> code that references it; for the virtual case, it (again, appears) to
go
> in to the section that contains a reference to a constructor. The
> specific situation in which this causes a problem is when the first
> constructor (for a class with a virtual destructor) appears in the
INIT
> (or other discardable) section, and reference to the virtual
destructor
> occurs after it is discarded. The particularly nasty (and not
> automatably addresable) complication of this is that this will occur
if
> any of the classes base classes (if any) satisfy the same conditions
and
> appear in the INIT section. There is no reasonable way to ensure that
> this does not occur, other than to avoid the use of virtual
destructors
> completely, or using the following, rather brutal hack, that, as I
> mentioned, I do not use (with a few specific exceptions that I have
> researched extensively; that being said, it is probably still not the
> best idea, but rather one that I have gotten away with, and very
> possibly will be able to continue to do so, at least in the very
> meaningful sense (in my particular case, which is unusual, but,
> unfortunately, I may not describe)). Additional fun can be had
because
> the compiler will generate destructors for scalar and vector deletion,
> and these, if unspecified and not worked around, will end up in INIT
> under the same conditions. OK, one more twist. The later only occurs
> for checked builds.
>
> A work around to this, such as it is, is that to do the best you can
to
> ensure this issue, for any class that uses a virtual destructor, and
the
> at least one constructor for this class will occur in a pageable
section
> (under the conditions described above), you must create another
> constructor (that may just be empty) that appears in the same source
> module as the one that will appear in the INIT section, and it must
> occur before that constructor. As mentioned above, but reiterated
here
> because it is very easy to overlook, this is true for any class that
> derives from a class which contains a virtual destructor. If all of
> this sounds very suspect, I would tell you that it seems to work, and
> then agree with you. The only confirmation, such as it is, that I can
> offer you, is that I came up with this independently of DriverWorks
(but
> not, by a long shot, on my own), and we seem to agree. Normally, this
> would be comforting; however, as has been pointed out many times in
this
> list (and I don’t think I have ever seen anyone disagree), DriverWorks
> is a proverbial Piece of Shit, so I don’t know what this really
offers.
>
> I’m sorry, but I can not offer you a concrete example that I know
> exhibits it, because of the nature of the work in which I am involved.
> This, however, is, to the best of my understanding and research,
> entirely accurate, including the part about it being labile.
>
> The situation for templates is much, much worse, in the sense of
being,
> without access to the source code for the compiler in question, there
is
> really no way (that I know of) to detail this conditions that dictate
> where any piece of instantiated templated code might appear.
Formally,
> the whole issue is a Halting problem, and, as many such issues due,
> almost assuredly does have an answer, but not one that can be relied
> upon, unless, of course, you happen to know it. Ironically, and some
> would, quite reasonably, say foolishly, I basically don’t use virtual
> destructors, and I do, quite causiously, use templates.
>
> The long and short of this is that, to be safe as possible, you have
to
> look at complete map file, searching for the particulars, taking in
to
> account the base class issue. This is very difficult, as well as time
> consuming, so I have to really, really want something to do it. I
have
> written a tool (or modified an existing tool) that does, reliably, it
> seems, process images using the symbols to verify that particular
> instances of this problem are not occuring. This, of course, only
works
> by my specifying accurately and without fail, what it should look for.
>
> I don’t imagine this helped very much, but this is, unfortunately, the
> best I have to offer. The best source of information I can point you
to
> on this, is the documentation for DriverWorks, if you have access to
it.
> I, presumably, may not include or note it, due to licensing issues.
> And, finally, as noted above, the product, well, sucks. The long and
> short is that, using these features of C++ (not to mention exception
> handling), needs to be carefully weighed against the potential
downside.
> In my case, which is, I think, unusual, I find it to be worth it.
>
> MM
>

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

55

I’ve disassembled my driver to check that everything was ok (Actually I
disassemble my drivers very often because I am a sick an dangerous pervert
but that’s not the point :p). If I’ve got time I’ll write a tool that
automatically check the presence of the autogenerated code in the
appropriate section, it’s doable because you can make signatures for such
code.

As for virtual functions I tend to avoid them in kernel mode. I’m also
cautious with smartpointers, because they imply a memory allocation
(tautology) which is not always wanted for performances reasons. I came
across that problem the other day when driver verifier exhibited a lot of
memory allocations/frees for one of my drivers, I was using a SmartPointer
in IRP_MJ_READ which was stupid of me since an object in the stack would do
the trick. On the other hand taking from the stack can lead to stack
exhaustion if your function is recursive. Decisions, dedicisions…

Now I’ll check my placement new operator problem with a fresh and rested
mind and let you know… If it’s a typo we’ll have a nice example of the
butterfly effect, “how a typo can generate a holy war”.

–

Edouard

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Doron Holan
Sent: Monday, April 24, 2006 04:34
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Re:Placement new operator in a driver

Autogenerated code lives all over the place, some you can avoid, some you
can’t. It includes scalar and vector deletes, both of which you cannot
write on your own. If the linker puts these in the wrong section, you are
hosed.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of cristalink
Sent: Sunday, April 23, 2006 4:05 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Re:Placement new operator in a driver

According to my experience, all code, including templates, goes to a default
segment, which is non-paged. If one starts to mess with autogenerated code,
such as constructors, destructors, copy constructors etc, they are looking
for troubles. There is not much point in having constructors or destructors
paged, as they are usually short, at least my ones.

the documentation for DriverWorks, if you have access to it

No, thank you :slight_smile:

–

“Martin O’Brien” wrote in message
news:xxxxx@ntdev…
>I haven’t looked at this problem in quite a while, as my experiences
>with them convinced me to avoid them where I was not willing to check
>out the generated code extensively. I can give you an example for
>virtual destructors; not for templates (I explain why later). The
basic
> problem with virtual destructors is the same as that with templates:
> code can end up in pageable segments, which can (and generally will)
> cause problems that I’m sure your familiar with. The significant
> difference is that the manifestations of the case of virtual
destructors
> can be, to a significant extent, predicted and worked around. That
> being said, I rarely do so, as it is cumbersome, ugly, quite possibly
> likely to break when the compiler changes, and, in the case of the
> kernel, avoidable by changing the representation. Normally, this is,
in
> my opinion, about the worst thing one can do; however, in this case,
the
> reasons are cogent. Unfortunately, as my principal motivation for
using
> C++ in the kernel (anywhere I roam, really) is templates, this is of
no
> help here.
>
> Please keep in mind that this is all empirical, and likely to change
> with the compiler.
>
> For every version of VC++ that I have looked at closely (which does
not
> include the most recent one, and, possibly the penultimate one as
well),
> the section in to which the code for a destructor is emitted depends
on
> whether it is virtual or not. In the case of a non virtual
destructor,
> the code (appears, at least) to go in to the same section as the first
> code that references it; for the virtual case, it (again, appears) to
go
> in to the section that contains a reference to a constructor. The
> specific situation in which this causes a problem is when the first
> constructor (for a class with a virtual destructor) appears in the
INIT
> (or other discardable) section, and reference to the virtual
destructor
> occurs after it is discarded. The particularly nasty (and not
> automatably addresable) complication of this is that this will occur
if
> any of the classes base classes (if any) satisfy the same conditions
and
> appear in the INIT section. There is no reasonable way to ensure that
> this does not occur, other than to avoid the use of virtual
destructors
> completely, or using the following, rather brutal hack, that, as I
> mentioned, I do not use (with a few specific exceptions that I have
> researched extensively; that being said, it is probably still not the
> best idea, but rather one that I have gotten away with, and very
> possibly will be able to continue to do so, at least in the very
> meaningful sense (in my particular case, which is unusual, but,
> unfortunately, I may not describe)). Additional fun can be had
because
> the compiler will generate destructors for scalar and vector deletion,
> and these, if unspecified and not worked around, will end up in INIT
> under the same conditions. OK, one more twist. The later only occurs
> for checked builds.
>
> A work around to this, such as it is, is that to do the best you can
to
> ensure this issue, for any class that uses a virtual destructor, and
the
> at least one constructor for this class will occur in a pageable
section
> (under the conditions described above), you must create another
> constructor (that may just be empty) that appears in the same source
> module as the one that will appear in the INIT section, and it must
> occur before that constructor. As mentioned above, but reiterated
here
> because it is very easy to overlook, this is true for any class that
> derives from a class which contains a virtual destructor. If all of
> this sounds very suspect, I would tell you that it seems to work, and
> then agree with you. The only confirmation, such as it is, that I can
> offer you, is that I came up with this independently of DriverWorks
(but
> not, by a long shot, on my own), and we seem to agree. Normally, this
> would be comforting; however, as has been pointed out many times in
this
> list (and I don’t think I have ever seen anyone disagree), DriverWorks
> is a proverbial Piece of Shit, so I don’t know what this really
offers.
>
> I’m sorry, but I can not offer you a concrete example that I know
> exhibits it, because of the nature of the work in which I am involved.
> This, however, is, to the best of my understanding and research,
> entirely accurate, including the part about it being labile.
>
> The situation for templates is much, much worse, in the sense of
being,
> without access to the source code for the compiler in question, there
is
> really no way (that I know of) to detail this conditions that dictate
> where any piece of instantiated templated code might appear.
Formally,
> the whole issue is a Halting problem, and, as many such issues due,
> almost assuredly does have an answer, but not one that can be relied
> upon, unless, of course, you happen to know it. Ironically, and some
> would, quite reasonably, say foolishly, I basically don’t use virtual
> destructors, and I do, quite causiously, use templates.
>
> The long and short of this is that, to be safe as possible, you have
to
> look at complete map file, searching for the particulars, taking in
to
> account the base class issue. This is very difficult, as well as time
> consuming, so I have to really, really want something to do it. I
have
> written a tool (or modified an existing tool) that does, reliably, it
> seems, process images using the symbols to verify that particular
> instances of this problem are not occuring. This, of course, only
works
> by my specifying accurately and without fail, what it should look for.
>
> I don’t imagine this helped very much, but this is, unfortunately, the
> best I have to offer. The best source of information I can point you
to
> on this, is the documentation for DriverWorks, if you have access to
it.
> I, presumably, may not include or note it, due to licensing issues.
> And, finally, as noted above, the product, well, sucks. The long and
> short is that, using these features of C++ (not to mention exception
> handling), needs to be carefully weighed against the potential
downside.
> In my case, which is, I think, unusual, I find it to be worth it.
>
> MM
>

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
----------------------------------------------------------------------------
-----------
Wanadoo vous informe que cet e-mail a ete controle par l’anti-virus mail.
Aucun virus connu a ce jour par nos services n’a ete detecte.

56

> Java and .NET got a syntax for locks

Bad idea. Very bad. I prefer any function calls or macros to the syntax
additions. Syntax additions are not flexible, and you cannot change the
underlying implementation and customize it for particular cases.

simply begs for a new language, at least with built-in notion of levels
of interruptability,

Again bad idea. Even C++ RTTI is bad idea, MFC’ DECLARE_DYNAMIC is better -
more flexible.

with built-in notion of pageability

…like #pragma code_seg(“PAGE”)

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

57

>>with built-in notion of pageability

…like #pragma code_seg(“PAGE”)
… which was added by MS so that the language could express some
new underlying reality.

In any case, my point was not about locks or RTTI being good or bad.
[I would agree though that semi-reflection provided by RTTI is inferior to DECLARE_DYNAMIC.]

The underlying reality has changed significantly, and at some point
adding crutches (pragmas, RTTI) is not enough.

Btw this sequential model shows its age not in the kernel only, think
about memory barriers, for example. You’d have to explain them to
those who wrote strtok 30-40 years ago.

You either have the syntax to express new semantics - or you don’t.

To use this syntax may of course be a matter of personal preferences.

Imho

lock(that) {… }

expresses the idea better than

{ CLock lock(&that); … }

because there’s no way to explain to the compiler that this last block
is kind of special (so hey compiler, please do not optimize this
“unused” lock variable away, will you?).

You are free to prefer { that.Lock(); …; that.Unlock(); }.

Some recent Alexandrescu’s and Sutter’s articles on the present and
future of C++ made me think that the move in this direction is at least
being thought over, yet again, I’m sure I will not see kernel written in D
or E or whatever in my lifetime.

----- Original Message -----
From: “Maxim S. Shatskih”
To: “Windows System Software Devs Interest List”
Sent: Monday, April 24, 2006 5:05 AM
Subject: Re: [ntdev] Re:Placement new operator in a driver

>> Java and .NET got a syntax for locks
> Bad idea. Very bad. I prefer any function calls or macros to the syntax
> additions. Syntax additions are not flexible, and you cannot change the
> underlying implementation and customize it for particular cases.
>
>> simply begs for a new language, at least with built-in notion of levels
>> of interruptability,
>
> Again bad idea. Even C++ RTTI is bad idea, MFC’ DECLARE_DYNAMIC is
> better -
> more flexible.
>
>>with built-in notion of pageability
>
> …like #pragma code_seg(“PAGE”)
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

58

> lock(that) {… }

What about the “unlock-wait-lock-retest_condition” pattern? rather frequent,
and not supported by this syntax.

What about:

Lock(container);
for each c in Container
{
if( IsOurObject(c) )
{
container.Detach(c);
Unlock(container);
return c;
}
}
Unlock(container);
return null;

  • again a very common find-and-detach pattern, and again not supported by the
    syntax (except using callbacks/delegates).

Alexandrescu is good, but he will never list all patterns we are meeting in our
real work.

Also looks like all these “pattern inventors” are not good friends with threads
:slight_smile: adding threads to their patterns introduces the new patterns not in their
books, like the 2 above one.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

59

> What about the “unlock-wait-lock-retest_condition” pattern? rather

frequent,
… as are dozens of others. I used lock as a lang elt just as an example.

Your (manual) solution is ok, but then some day someone inherits the code
and inserts one more “return” in the loop, and, being only a human under
stress,
forgets to detach, or to unlock, or both…

What’s wrong with semi-automating the process?

Let the dtor do what you need on each and every return/break
and live happily ever after, let’s put the silicon chip (the compiler)
at work - if and when possible.

Anyway, I still think it was not the point.

As was shown (by Jan), lang shortcuts may simplify life;
as was shown (by Doron), they may introduce just one
seemingly innocuous “a = b;” or “a = b->qqq()” you spend
a night with.

Look, we had C.

Then comes a guy with a piece of hw (I omit 16 chapters), and
we get “volatile” qualifier. New reality - new language elts.

Then “++ctr” becomes risky because ctr is shared, and the hw guy
invents InterlockedXYZ for us. Oh, we got thru with lib changes
this time - ok, fine.

Now welcome to the kernel (or, to put it another way, to highly
optimized multithreaded programming with interruptability levels).

Can I get a compiler error by waiting in a handler that may be
called at dispatch? No, the language does not know anything.

As was shown, the compiler is so eager to optimize that it may
even factor out different accessors (at a cost of two lonely nights),
and all pragmas in the world can not prevent it.

The compiler has been taught to know “is a” vs “has a”?
Good, but not good enough.
Can we get away with lib changes and pragmas?
Well, so far we simply have to.

I was thinking (dreaming) about a compiler that knows.

C was invented because algol and fortran did not know
something:-)

----- Original Message -----
From: “Maxim S. Shatskih”
To: “Windows System Software Devs Interest List”
Sent: Monday, April 24, 2006 11:31 AM
Subject: Re: [ntdev] Re:Placement new operator in a driver

>> lock(that) {… }
>
> What about the “unlock-wait-lock-retest_condition” pattern? rather
> frequent,
> and not supported by this syntax.
>
> What about:
>
> Lock(container);
> for each c in Container
> {
> if( IsOurObject(c) )
> {
> container.Detach(c);
> Unlock(container);
> return c;
> }
> }
> Unlock(container);
> return null;
>
> - again a very common find-and-detach pattern, and again not supported by
> the
> syntax (except using callbacks/delegates).
>
> Alexandrescu is good, but he will never list all patterns we are meeting
> in our
> real work.
>
> Also looks like all these “pattern inventors” are not good friends with
> threads
> :slight_smile: adding threads to their patterns introduces the new patterns not in
> their
> books, like the 2 above one.
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

60

Ok so what happened this morning is that I added that placement new operator
and evertyhing went well. Thanks for your help. The problem was caused by a
typo in inheritance, my apologies for all this… Much ado about nothing…

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Edouard A.
Sent: Monday, April 24, 2006 08:28
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Re:Placement new operator in a driver

I’ve disassembled my driver to check that everything was ok (Actually I
disassemble my drivers very often because I am a sick an dangerous pervert
but that’s not the point :p). If I’ve got time I’ll write a tool that
automatically check the presence of the autogenerated code in the
appropriate section, it’s doable because you can make signatures for such
code.

As for virtual functions I tend to avoid them in kernel mode. I’m also
cautious with smartpointers, because they imply a memory allocation
(tautology) which is not always wanted for performances reasons. I came
across that problem the other day when driver verifier exhibited a lot of
memory allocations/frees for one of my drivers, I was using a SmartPointer
in IRP_MJ_READ which was stupid of me since an object in the stack would do
the trick. On the other hand taking from the stack can lead to stack
exhaustion if your function is recursive. Decisions, dedicisions…

Now I’ll check my placement new operator problem with a fresh and rested
mind and let you know… If it’s a typo we’ll have a nice example of the
butterfly effect, “how a typo can generate a holy war”.

–

Edouard

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Doron Holan
Sent: Monday, April 24, 2006 04:34
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Re:Placement new operator in a driver

Autogenerated code lives all over the place, some you can avoid, some you
can’t. It includes scalar and vector deletes, both of which you cannot
write on your own. If the linker puts these in the wrong section, you are
hosed.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of cristalink
Sent: Sunday, April 23, 2006 4:05 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Re:Placement new operator in a driver

According to my experience, all code, including templates, goes to a default
segment, which is non-paged. If one starts to mess with autogenerated code,
such as constructors, destructors, copy constructors etc, they are looking
for troubles. There is not much point in having constructors or destructors
paged, as they are usually short, at least my ones.

the documentation for DriverWorks, if you have access to it

No, thank you :slight_smile:

–

“Martin O’Brien” wrote in message
news:xxxxx@ntdev…
>I haven’t looked at this problem in quite a while, as my experiences
>with them convinced me to avoid them where I was not willing to check
>out the generated code extensively. I can give you an example for
>virtual destructors; not for templates (I explain why later). The
basic
> problem with virtual destructors is the same as that with templates:
> code can end up in pageable segments, which can (and generally will)
> cause problems that I’m sure your familiar with. The significant
> difference is that the manifestations of the case of virtual
destructors
> can be, to a significant extent, predicted and worked around. That
> being said, I rarely do so, as it is cumbersome, ugly, quite possibly
> likely to break when the compiler changes, and, in the case of the
> kernel, avoidable by changing the representation. Normally, this is,
in
> my opinion, about the worst thing one can do; however, in this case,
the
> reasons are cogent. Unfortunately, as my principal motivation for
using
> C++ in the kernel (anywhere I roam, really) is templates, this is of
no
> help here.
>
> Please keep in mind that this is all empirical, and likely to change
> with the compiler.
>
> For every version of VC++ that I have looked at closely (which does
not
> include the most recent one, and, possibly the penultimate one as
well),
> the section in to which the code for a destructor is emitted depends
on
> whether it is virtual or not. In the case of a non virtual
destructor,
> the code (appears, at least) to go in to the same section as the first
> code that references it; for the virtual case, it (again, appears) to
go
> in to the section that contains a reference to a constructor. The
> specific situation in which this causes a problem is when the first
> constructor (for a class with a virtual destructor) appears in the
INIT
> (or other discardable) section, and reference to the virtual
destructor
> occurs after it is discarded. The particularly nasty (and not
> automatably addresable) complication of this is that this will occur
if
> any of the classes base classes (if any) satisfy the same conditions
and
> appear in the INIT section. There is no reasonable way to ensure that
> this does not occur, other than to avoid the use of virtual
destructors
> completely, or using the following, rather brutal hack, that, as I
> mentioned, I do not use (with a few specific exceptions that I have
> researched extensively; that being said, it is probably still not the
> best idea, but rather one that I have gotten away with, and very
> possibly will be able to continue to do so, at least in the very
> meaningful sense (in my particular case, which is unusual, but,
> unfortunately, I may not describe)). Additional fun can be had
because
> the compiler will generate destructors for scalar and vector deletion,
> and these, if unspecified and not worked around, will end up in INIT
> under the same conditions. OK, one more twist. The later only occurs
> for checked builds.
>
> A work around to this, such as it is, is that to do the best you can
to
> ensure this issue, for any class that uses a virtual destructor, and
the
> at least one constructor for this class will occur in a pageable
section
> (under the conditions described above), you must create another
> constructor (that may just be empty) that appears in the same source
> module as the one that will appear in the INIT section, and it must
> occur before that constructor. As mentioned above, but reiterated
here
> because it is very easy to overlook, this is true for any class that
> derives from a class which contains a virtual destructor. If all of
> this sounds very suspect, I would tell you that it seems to work, and
> then agree with you. The only confirmation, such as it is, that I can
> offer you, is that I came up with this independently of DriverWorks
(but
> not, by a long shot, on my own), and we seem to agree. Normally, this
> would be comforting; however, as has been pointed out many times in
this
> list (and I don’t think I have ever seen anyone disagree), DriverWorks
> is a proverbial Piece of Shit, so I don’t know what this really
offers.
>
> I’m sorry, but I can not offer you a concrete example that I know
> exhibits it, because of the nature of the work in which I am involved.
> This, however, is, to the best of my understanding and research,
> entirely accurate, including the part about it being labile.
>
> The situation for templates is much, much worse, in the sense of
being,
> without access to the source code for the compiler in question, there
is
> really no way (that I know of) to detail this conditions that dictate
> where any piece of instantiated templated code might appear.
Formally,
> the whole issue is a Halting problem, and, as many such issues due,
> almost assuredly does have an answer, but not one that can be relied
> upon, unless, of course, you happen to know it. Ironically, and some
> would, quite reasonably, say foolishly, I basically don’t use virtual
> destructors, and I do, quite causiously, use templates.
>
> The long and short of this is that, to be safe as possible, you have
to
> look at complete map file, searching for the particulars, taking in
to
> account the base class issue. This is very difficult, as well as time
> consuming, so I have to really, really want something to do it. I
have
> written a tool (or modified an existing tool) that does, reliably, it
> seems, process images using the symbols to verify that particular
> instances of this problem are not occuring. This, of course, only
works
> by my specifying accurately and without fail, what it should look for.
>
> I don’t imagine this helped very much, but this is, unfortunately, the
> best I have to offer. The best source of information I can point you
to
> on this, is the documentation for DriverWorks, if you have access to
it.
> I, presumably, may not include or note it, due to licensing issues.
> And, finally, as noted above, the product, well, sucks. The long and
> short is that, using these features of C++ (not to mention exception
> handling), needs to be carefully weighed against the potential
downside.
> In my case, which is, I think, unusual, I find it to be worth it.
>
> MM
>

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
----------------------------------------------------------------------------
-----------
Wanadoo vous informe que cet e-mail a ete controle par l’anti-virus mail.
Aucun virus connu a ce jour par nos services n’a ete detecte.

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
----------------------------------------------------------------------------
-----------
Wanadoo vous informe que cet e-mail a ete controle par l’anti-virus mail.
Aucun virus connu a ce jour par nos services n’a ete detecte.