C++ RTL for NT kernel-mode drivers

> If you want to use a proper OO language with rich run-time

support, pray for the day that we can use managed code
in kernel-mode… and don’t expect a solution soon.

It’s will be great… and the whole ‘dbg_console’ instead the current DbgPrint() to write there commands and scripts for the kebash and kepython -)

It seems that every few years somebody surfaces with
a new one of these things.
Didn’t we see something like this, with one particular
dev posting with great fervor, a few years back?

And the people who try to use C++ in the kernel and get stuck bad are occur a bit more often. And it supposed to be a good deal if those people are able to get the code, see it and make a decision a kind of “oh, EH in kernel is a peace of junk indeed” or “hmm, maybe it worth try for something”, isn’t it ?
If you mean ‘one particular dev’ Peter Hurley, than i’v seen his library with a great interest.

While cute, and an interesting personal learning exercises,
they’re never suitable for commercial products.

Yes, it was certainly the very interesting development.
And now every person got this code can make decision about it’s using on her own preferences. Is there something wrong ? If some code exists it’s a bit better than otherwise. -)

Wrong? Not so very wrong, no.

See… I’ve personally never believed that. That’s one reason why I’ve never been an adherent of the OSS movement. “Here’s a lot of code that, while it works very well in some aspects, is shit in other aspects. Please figure out for yourself which parts of the code are which. But it’s free and you can use it. And we had fun writing the parts that we wrote. You’re welcome.”

From my own little viewpoint, having code available to the developers that is problematic and/or does not follow best practices, is worse than having no code available. This type of code is just a big hole for unsuspecting devs to fall into.

It’s hard enough to write Windows drivers without having to deal with additional unknowns.

This isn’t meant as a specific criticism of your endeavour. Rather, it’s my view on the whole genre.

Peter
OSR

> If you want to use C++ for driver development (I think that’s too bad, but) you have whatever

Microsoft now supports.

Stack size is one of the issues.

If somebody will really port STL/boost to kmode - then good luck deal with all these by-value objects.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

>From my own little viewpoint, having code available to the developers that
is problematic and/or does not follow best practices, is worse than having
no code available. This type of code is >just a big hole for unsuspecting
devs to fall into.

Tru dat.

And if it’s a commonly available bad example, it takes on a life of its own
after a while.

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@osr.com
Sent: Tuesday, November 12, 2013 9:43 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] C++ RTL for NT kernel-mode drivers

Wrong? Not so very wrong, no.

[quote]
If some code exists it’s a bit better than otherwise. -) [/quote]

See… I’ve personally never believed that. That’s one reason why I’ve
never been an adherent of the OSS movement. “Here’s a lot of code that,
while it works very well in some aspects, is shit in other aspects. Please
figure out for yourself which parts of the code are which. But it’s free
and you can use it. And we had fun writing the parts that we wrote. You’re
welcome.”

From my own little viewpoint, having code available to the developers that
is problematic and/or does not follow best practices, is worse than having
no code available. This type of code is just a big hole for unsuspecting
devs to fall into.

It’s hard enough to write Windows drivers without having to deal with
additional unknowns.

This isn’t meant as a specific criticism of your endeavour. Rather, it’s my
view on the whole genre.

Peter
OSR


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

On 11/12/2013 5:43 PM, xxxxx@osr.com wrote:

See… I’ve personally never believed that. That’s one reason why I’ve never been an adherent of the OSS movement. “Here’s a lot of code that, while it works very well in some aspects, is shit in other aspects. Please figure out for yourself which parts of the code are which. But it’s free and you can use it. And we had fun writing the parts that we wrote. You’re welcome.”

Isn’t it more like “Here’s a lot of code. If it doesn’t work we’ll do
our best to fix it if we have the time and energy, otherwise you have
the source to figure out a fix or workarounds yourself”?


Bruce Cran

On Tue, Nov 12, 2013 at 1:18 PM, Bruce Cran wrote:

> On 11/12/2013 5:43 PM, xxxxx@osr.com wrote:
>
>> See… I’ve personally never believed that. That’s one reason why I’ve
>> never been an adherent of the OSS movement. “Here’s a lot of code that,
>> while it works very well in some aspects, is shit in other aspects. Please
>> figure out for yourself which parts of the code are which. But it’s free
>> and you can use it. And we had fun writing the parts that we wrote.
>> You’re welcome.”
>>
>
> Isn’t it more like “Here’s a lot of code. If it doesn’t work we’ll do our
> best to fix it if we have the time and energy, otherwise you have the
> source to figure out a fix or workarounds yourself”?

As I work with an open source team of developers, this isn’t really what is
going on. “Works” modulo the usual bugs in any complex software product, is
on one branch, “Works for shit but has new features” is on another, the
development branch. We get to leverage the efforts of many and in general
the release branches of the big projects (Linux for example) are rather
high quality. The dev branches not so much, but that is a known risk
factor.

That is the good news. The real problem is the churn in the release branch.
Because the dev branch is loose and has people adding cool features etc.
shit that works just fine regularly gets re-implemented for arbitrary
reasons. Then that stuff moves to the release branch and people using that
branch to add value via their own products get to deal with the arbitrary
churn.

More specifically people with patch queues on release branches frequently
get to toss all their nice changes and enhancements out and re-implement
them, refix re-introduced bugs, etc. all over again. And again. And again.
And fight with the reigning authorities of this product or that product to
get their conflicting features into mainline and out of patch status rather
than somebody else’s conflicting features.

Mark Roddy

Generally, I avoid free software for one important reason: I can’t afford it.

Nobody would pay me to fix a bug i gcc. And can I really ship a product
to a client who can’t recompile te source without my modified compier.

I worked for many years with an optimizing compiler. It had not achieved
“critical mass” (a software entity is said to have reached “critical mass”
when fixing one bug introduces 1+ epsilon bugs), but there was a high
probability that fixing one bug would either introduce or expose one other
bug; it was in delicate balance.

As a consequence, we could never recompile our operating system from
source overnight; formerly-working components of the OS would likely fail
because some ew bug would be found. It was a regular occurrence that
while working on component X that a point release/hotfix equivalent of the
compier would itroduce a bug in formerly-working code. Because we worked
closely with the compiler writers, and many (like me) were already
compiler eeks, we would get te details of what the “root cause” was, for
example, certain kinds of potential aliasing would not be detected as
invalidating a common subexpression, so a “stale” result might be reused.
Tweaking the guts of an optimizing compiler is not for te faint of heart.

So if you find that one of the parts of your open-source component is
broken, yes, feel free to fix it yourself. And if the basement of this
new house you are considering is too small, feel free to just dig a
subbasement. Don’t worry if you don’t understand basic structural
principles, hey, the worst that can happen is that your house can
collapse.

This is not to say that all of Windows is beautiful code, exemplary in
every way of Best Practice. But I’ve seen horrible code, and when I
criticized it, I was told “But I learned how to do that from open-source system here>, and that’s the clever [editorial comment here:
add incomprehensible and unmaintainable] way it’s done there”. Sort of
the same crappy quality of many MSDN examples, whose quality has doomed
several real projects; I saw one $500,000 investment fail because the
author used te MSDN multi-threaded async socket example code (other than
the example got networking wrong, threading wrong, and synchronization
wrong, there was nothing wrong with the example).

Even the WDM examples were often poor examples, since there were huge
numbers of “clever” tricks used that resulted in unmaintainable code
(IoSkipCurrentIrpStackLocation hacks, for example, that improved
performance on a 386 but had no noticeable effect on Pentium 4+ machines)
or left readers confused as to why it was really done. And the horrors
of WDM power management are the best argument for KMDF I have ever seen.

Visible source does not work as well as the free-open-source people like
to believe.

And the other issue is te GNU license, which is one of the biggest
barriers to code-sharing that ever existed, and I think has done far more
harm than good. My clients didn’t want anything with a GNU license
anywhere near their product code; if they could have kept every trace of
such code off my machine (fearing cross-contamination) they would have
done so. Nobody ever lost by underestimating the paranoia of corporate
legal staffs. Or their ability to overreact to the phrase “open source”
(never mind that BSD, Boost, CodeProject, and Creative Commons have sane
license: all open-source is Evil).

Sorry to head to flame-war, Peter, but you did open the door… (move
reactions to nttalk, which I don’t get, and therefore will be relieved of
any compulsion to reply)
joe

>


>
> Wrong? Not so very wrong, no.
>
>


>
> See… I’ve personally never believed that. That’s one reason why I’ve
> never been an adherent of the OSS movement. “Here’s a lot of code that,
> while it works very well in some aspects, is shit in other aspects.
> Please figure out for yourself which parts of the code are which. But
> it’s free and you can use it. And we had fun writing the parts that we
> wrote. You’re welcome.”
>
> From my own little viewpoint, having code available to the developers that
> is problematic and/or does not follow best practices, is worse than having
> no code available. This type of code is just a big hole for unsuspecting
> devs to fall into.
>
> It’s hard enough to write Windows drivers without having to deal with
> additional unknowns.
>
> This isn’t meant as a specific criticism of your endeavour. Rather, it’s
> my view on the whole genre.
>
> Peter
> OSR
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

Well… I guess it depends. Isn’t it really “Here’s a lot of code. If it doesn’t work we’ll FIX IT IF WE HAPPEN TO FIND THAT PART OF THE PROJECT AMUSING AND OF USE TO US otherwise you have
the source to figure out a fix or workarounds yourself”?

And thus, in many cases, you’d be better off starting from the beginning yourself.

Give me a closed source process, where engineering discipline is enforced during development, ANYday.

Mr. Roddy addresses the “big project” OSS world. This isn’t the flavor I was specifically thinking of… rather, I was thinking of more “personal-sized” OSS projects… such as that by the OP who said “better some software than no software”

There are many different flavors of OSS, and they each have their own particular attributes:

  • The “big project” world that Mr. Roddy addressed.

-The “personal-sized” OSS projects like that of the OP who effectively said “better some software than no software.” This is the flavor I was specifically addressing.

  • The “consortium-based” OSS project, where an industry group gets together to build a reference model of something.

Each flavor is its own special world. I’m sure there are other flavors.

Again, I was speaking really to the “personal-project sized” OSS flavor of project, like the OPs. Stuff like Linux, well… I can’t hardly comment on that with any experience. For that I’d have to wait for Anton to tell me what to say. For the “consortium-based” OSS projects… I really don’t have anything positive at all to say except that it is an excellent activity for somebody who wants to exercise their political strategy skills.

Peter
OSR

On 11/12/2013 8:24 PM, xxxxx@osr.com wrote:

Well… I guess it depends. Isn’t it really “Here’s a lot of code. If it doesn’t work we’ll FIX IT IF WE HAPPEN TO FIND THAT PART OF THE PROJECT AMUSING AND OF USE TO US otherwise you have
the source to figure out a fix or workarounds yourself”?

And thus, in many cases, you’d be better off starting from the beginning yourself.

Give me a closed source process, where engineering discipline is enforced during development, ANYday.

To see the sort of discipline involved in Linux-type projects, take a
look at http://www.freebsd.org/releases/10.0R/schedule.html . In a
previous job at a large tech company I was left running a continuous
integration server under my desk while people sent check-in notices
manually via email and argued they didn’t have time to write unit
tests. In contrast FreeBSD runs a tinderbox continuously, and people
care enough about the project as a whole that they make efforts to do
things properly.

On the other hand, I took over a Windows driver a few years ago, made
some bug fixes and a few releases - and haven’t had time to keep it up.
I know it has some major bugs and I have people offering me money to fix
it, but I simply don’t have time to work on it. For the people who use
it it’s little better than a closed-source product from a company that’s
gone out of business, since few people have the knowledge to take it
over and fix it.


Bruce Cran

> [quote]

Isn’t it more like “Here’s a lot of code. If it doesn’t work we’ll do
our best to fix it if we have the time and energy, otherwise you have
the source to figure out a fix or workarounds yourself”?
[/quote]

Well… I guess it depends. Isn’t it really “Here’s a lot of code. If it doesn’t
work we’ll FIX IT IF WE HAPPEN TO FIND THAT PART OF THE PROJECT
AMUSING AND OF USE TO US otherwise you have
the source to figure out a fix or workarounds yourself”?

And thus, in many cases, you’d be better off starting from the beginning
yourself.

Give me a closed source process, where engineering discipline is enforced
during development, ANYday.

We are seeing regular BSoD’s at a client with Win2008R2. I posted on the ms forums and got a lot of “me too”'s. Apart from try and wave and get Microsofts attention there is not a lot else I can do but hope that they find the bug significant enough to bother fixing. It’s a rather helpless feeling.

Same with a 2003 bug a while back. It took ages for Microsoft to get around to fixing it, and it caused a big waste of time while employees hung around waiting for the system to boot up again.

James

>

It seems that every few years somebody surfaces with a new one of these
things. Didn’t we see something like this, with one particular dev posting
with great fervor, a few years back?

While cute, and an interesting personal learning exercises, they’re never
suitable for commercial products.

If you want to use C++ for driver development (I think that’s too bad, but)
you have whatever Microsoft now supports.

If you want to use a proper OO language with rich run-time support, pray for
the day that we can use managed code in kernel-mode… and don’t expect a
solution soon.

Under Linux you can use FUSE which allows you to have a filesystem implemented in userspace. It turns out that the performance overheads of doing so aren’t significant enough to bother implementing the filesystem in the kernel.

For a network based distributed storage system implemented in C++, I’ve wondered if the same approach could be taken under Windows - a kernel driver (eg storport) to pass requests and responses between kernel and userspace, but the actual work implemented in userspace. Obviously you couldn’t use pagefile, hibernate, etc, but for regular file/database/etc storage, would the performance overheads be significant enough to worry about with the nt kernel?

James

LOL… you know, James, I really do respect what you do. But these types of posts really DO make me laugh.

So there’s some obscure bug in Windows. Do you *really* have the time to invest to learn enough about the Windows source code to confidently find and fix this sort of problem? And if I’m a customer, do you seriously think I’d be willing to take your solution and run it on my production server?

You have a bug in Windows, open an incident or contact your TAM. Bug repro’ed, QFE built and issued, done. I don’t know abaout you, but I have waaaay better things to do with my time than fix somebody ELSE’s code.

I remember THE first day I had access to Windows source code. This was *many* (maybe 15?) years ago. The very first thing I went to look up was a question that I had since before Windows NT 3.1 was released and that nobody could answer to my satisfaction: When I want to pend an IRP, why do I have to BOTH mark the IRP pending *and* return STATUS_PENDING.

Well, it took me about 2 years of working with Windows source code to be able to answer that question with authority.

Maybe I’m just slow.

We (OSR) actually have a kit that lets you write file systems in user mode. It’s a “limited availability” thing because we don’t want people to misuse it.

There are a number of tricky issues, but the performance is surprisingly decent.

Aside from file systems, consider the fact that UMDF lets people write drivers in user mode. For many devices, even high speed USB devices with bulk pipes, the throughput is more than acceptable.

Peter
OSR

So that you feel in a position to blame me for OT discussion on NTDEV? To be honest, I think this entire thread
firmly belongs on NTTALK right from the original post…

In any case, it is pretty amusing to see how C+±related topic slowly morphs into GPL-related topics, Linux, FOSS, etc,etc,etc. What I am waiting for now is Mr.Kyler jumping in and moving this discussion even further off-topic, i.e to the domain of political science…

Anton Bassov

Well, we *have* veered a bit off-course, so there’s been some thread drift.

But the topics mostly HAVE been about Windows system software development.

Even my whinging about OSS was related to OSS for Windows.

Peter
OSR

Doron, I for one certainly appreciate your hard work in getting the compiler group to support the /kernel option. I’ve personally written a ton of C++ kernel code over the years, and it’s nice to have a compiler switch that embodies the previously informal rules.

Thanks!
Jan

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Doron Holan
Sent: Monday, November 11, 2013 4:33 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] C++ RTL for NT kernel-mode drivers

I made /kernel happen…so blame me.if you want. It has nothing to do with excluding others from a potential market, rather it is codifying the informal rules as formal. EH is not the future in the kernel as far as MSFT is concerned, the danger of misuse and corrupting system state is great. If that changes, we will update the flag to allow it as appropriate

d

Well kt133a, if you had just put this in the first post instead of buried deep in the thread:

"It hardly beleived this RTL can be considered as a part of some commercial
product in some future. Let’s just look at the current state of the project:

  1. this is entirely a private investigative initiative;
  2. a whole bunch of undocumented or poorly-observed or reverse-engineered
    technologies is used,
  3. the officially unsupporetd or unrecommended facilities are used,
  4. initially released with the intention to be of an interest to the enthusiast
    of the C++ using in the NT kernel environment.

I doubt anyone would have even bothered responding to the initial post if it said that. We’ve had various EH usage in the kernel since the Walter Oney days and this particular effort really isn’t going in the right direction. Maybe instead it would be better to contact the provider of the best EH library out there and work with them to bring it forward instead of trying to re-invent the wheel and coming up a little short. I recall from one EH library that stack usage is now a non-issue.

> To see the sort of discipline involved in Linux-type projects, take a

look at http://www.freebsd.org/releases/10.0R/schedule.html

FreeBSD is not Linux-type

FreeBSD is a cathedral

Linux is a bazaar

:slight_smile:


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

>do I have to BOTH mark the IRP pending *and* return STATUS_PENDING.

Rajeev Nagar had an excellent explanation of this in his FS book, circa 1998.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

> I doubt anyone would have even bothered responding to the

initial post if it said that. We’ve had various EH usage
in the kernel since the Walter Oney days and this
particular effort really isn’t going in the right direction.
Maybe instead it would be better to contact the provider
of the best EH library out there and work with them to
bring it forward instead of trying to re-invent the
wheel and coming up a little short. I recall from one
EH library that stack usage is now a non-issue.

It seems to look not such a kind of pessimistic. The current library is modular enough internally for the EH-stuff to be quickly substituted by some more advanced implementation. On you courteousy you can post here some details about the lib mentioned, this is really interesting.

From my own little viewpoint, having code available to the
developers that is problematic and/or does not follow
best practices, is worse than having no code available.
This type of code is just a big hole for unsuspecting
devs to fall into.

This viewpoint IMO looks a bit like “let’s don’t give the forks to that barbarians for them not to prick out their eyes”. I lack this opinion but just can understand you may have solid argumentation for this e.g. similar as posted above the topic.

It’s hard enough to write Windows drivers without having
to deal with additional unknowns.

This is entirely the true. And in addition the c++ is a PL to be considered as a complex thing enough. But it hardly supposed the people involved in these activities are uncapable to see the pitfalls supplied.
So now when the most misconceptions (added mainly by me, because of it has became a bit unexpected that so many readers take this project as production ready) are probably cleared, can we lead to achieve the goals implicetly presumed as the starting reasons of this topic? Namely those goals are: to make the project available for most curious people as possible, to discuss the internal mechanics for the optimizations to get appeared and bugs to get disappered. It’s believed the project in spite of the well-known restrictions and admissions appears to be viable enough to be of interest for even if a bit of NT-kernel community besides me. -)

Oh BTW, thanks, Mr. Viscarola, the ‘who effectively said “better some software than no software”’ - now it’s me ;-).

> post here some details about the lib mentioned, this is really interesting.

Happy reading:

https://www.osronline.com/ShowThread.cfm?link=203234