About UCRT and KM/CRT

Oh, that’s pretty hard… I’m trying to understand how the C standard is implemented in the kernel, and what justifies the existence of km/crt in the wdk, why can’t I use headers from ucrt in the kernel while doing the implementation for the kernel? All this ‘defs hell’ and the fact that libcntpr.lib does not implement everything that is declared in the headers (malloc, open, clock, etc)… Why can’t I make a ucrt with a kernel and user implementation? Or am I just missing something?

well, if you want to know all of that, you better be prepared to fork over a bottle of scotch or two

remember that the C language is a standard for converting ‘high level’ code into machine language on a specific platform. Here we deal mostly with x64, x86, ARM and that ITANIUM thingy. but the C language has been ported to hundreds of other platforms including the ones that run the stop lights in you neighborhood most likely

but of course the CRT is another matter. Programs that tell you to walk or don’t walk, usually don’t need access to cosines, tangents or psudo-random numbers. They also don’t really need dynamic memory allocation either. The same is true of the programs that control the breaks on your car and the ventilators in the hospital.

KM programs in Windows also have to deal with a subset - although not nearly as restrictive as in some other environments

Why can’t I make a ucrt with a kernel and user implementation?

That is EXACTLY what libcntpr.lib is – the kernel implementation of ucrt. The things that aren’t implemented are things that need more information (malloc – you need to specify what pool to use), things that do not allow for appropriate error handling (open/fopen and friend), and other things that a kernel driver shouldn’t be doing anyway. A kernel driver is supposed to be little more than glue. You shouldn’t be writing applications in kernel mode.

@Tim_Roberts said:

Why can’t I make a ucrt with a kernel and user implementation?

That is EXACTLY what libcntpr.lib is – the kernel implementation of ucrt. The things that aren’t implemented are things that need more information (malloc – you need to specify what pool to use), things that do not allow for appropriate error handling (open/fopen and friend), and other things that a kernel driver shouldn’t be doing anyway. A kernel driver is supposed to be little more than glue. You shouldn’t be writing applications in kernel mode.

what about _beginthreadex(), what about clock() and others - what are the fundamental problems to implement them?

@MBond2 said:
well, if you want to know all of that, you better be prepared to fork over a bottle of scotch or two

remember that the C language is a standard for converting ‘high level’ code into machine language on a specific platform. Here we deal mostly with x64, x86, ARM and that ITANIUM thingy. but the C language has been ported to hundreds of other platforms including the ones that run the stop lights in you neighborhood most likely

but of course the CRT is another matter. Programs that tell you to walk or don’t walk, usually don’t need access to cosines, tangents or psudo-random numbers. They also don’t really need dynamic memory allocation either. The same is true of the programs that control the breaks on your car and the ventilators in the hospital.

KM programs in Windows also have to deal with a subset - although not nearly as restrictive as in some other environments

ok, but what is the reason that many headers are duplicated in km/crt from ucrt, while many are added (crtdefs.h, etc), which violate the general semantics of ucrt, which makes it impossible to use third-party implementations of some things based on standard headers from ucrt? When I connect the <time.h> I expect all functions from it to be implemented, otherwise, why are they defined there at all?

I’m not going to defend every choice. The NT kernel was simply not designed to run arbitrary general-purpose applications. They provided routines that are useful and safe. Kernel code needs to be extremely careful and handle error conditions properly. The CRT routines encourage you to forget about all of that.

2 Likes

what are the fundamental problems to implement them?

Well, there aren’t necessarily any fundamental problems implementing these… but they are user mode APIs with user mode conventions and constraints… and these have kernel mode equivalents that have their own, specific, kernel mode conventions and constraints. There’s just no reason to force-fit user mode conventions into the OS. Just compare _beginthreadex() to PsCreateSystemThread(). Why use a sledge hammer to make the former into the latter? I mean… who would want to use this?

Peter

1 Like

@“Peter_Viscarola_(OSR)” said:

what are the fundamental problems to implement them?

Well, there aren’t necessarily any fundamental problems implementing these… but they are user mode APIs with user mode conventions and constraints… and these have kernel mode equivalents that have their own, specific, kernel mode conventions and constraints. There’s just no reason to force-fit user mode conventions into the OS. Just compare _beginthreadex() to PsCreateSystemThread(). Why use a sledge hammer to make the former into the latter? I mean… who would want to use this?

Peter

The versatility of the interface? Implementation of the C standard? The ability to smoothly change the implementation? Simplifying the introduction of modern programming ideas in the kernel?

Вопрос был в другом :> @Tim_Roberts said:

I’m not going to defend every choice. The NT kernel was simply not designed to run arbitrary general-purpose applications. They provided routines that are useful and safe. Kernel code needs to be extremely careful and handle error conditions properly. The CRT routines encourage you to forget about all of that.

The question was asked in general : why are headers and definitions duplicated in the UCRT folder and KM/CRT folder and the same functionality is not provided? How do I find out what exactly is implemented from the malloc.h, stdlib.h, etc. headers in libcntpr.lib, and what is not? Then why are there all these headers and definitions that I can connect to the project, but I can’t use them?

This is the kernel, not some usermode application that any developer with half an IQ straight out of college (where he learned nothing as well) can write. Get over it. The last post of Peter is a good response to your “problem” already. I’d have liked to also talk about the privilege levels, the memory range (for example accessing the shared user data region), the pain in the ass to merge every changes done to the usermode API and add its kernel counterpart, the fact that most usermode apis DEPEND on the kernel apis, etc… but I sort of don’t feel like it, considering this thread shouldn’t even exist. But coming from someone who literally asked the following question on this forum: “How many spinlocks can I create?”; Let’s say that I’m not surprised.

The versatility of the interface? Implementation of the C standard? The ability to smoothly change the implementation? Simplifying the introduction of modern programming ideas in the kernel?

Programming in user-mode is not the same as programming in kernel-mode. And, while they share many of the same concerns, they do not share others. Like… threads. And… time. The issues you need to deal with in user-mode and in kernel-mode are very different.

Did you not look at the differences between _beginthreadex and PsCreateSystemThread? _beginthreadex was designed for user-mode use. It meets user-mode needs, and provides control for user-mode types of concerns. Likewise PsCreateSystemThread for kernel-mode. It’s a matter of the right tool for the right job. In this particular case, it’s not a matter of “standards” or “modern”… it’s a matter of suitability for purpose. I mean… you might just as well ask why _beginthreadex doesn’t allow the option to provide a handle to a process into which to create the thread or an access mask or an OBJECT_ATTRIBUTES pointer. It’s because that’s not the point of the function. But these things are vital when you’re running in kernel-mode, where the context in which you’re running may be specific or arbitrary.

You might as well, foolishly, ask why you can’t call malloc in kernel-mode. What could we possibly map that to that would make sense? The answer: Nothing. Because malloc makes sense in user-mode, but not in kernel-mode, where we have the concept of paged vs unpaged, executable vs nx, memory (for just a couple of tiny differences).

Having said that… if you want to call _beginthreadex in your kernel-mode code, and you can envision a set of configuration and constraint decisions that fit your specific requirements, nothing says you can’t implement your own version of _beginthreadex. I think that’d be a really bad idea… but it’d be a heck of a lot better than having some member of the kernel team make those configuration and constraint decisions for you, so that they probably wouldn’t fit your needs (or anybody else’s needs either).

The essence of engineering is creating suitable solutions that are fit for their purposes. NOT trying to force-fit the same paradigm into environments where that paradigm makes little sense.

Peter

The versatility of the interface? Implementation of the C standard.

_beginthreadex is itself a Microsoft extension. The C standard library has no notion of threads, processes or even directories. And if you believe that _beginthreadex is more versatile than PsCreateSystemThread, then you have clearly never looked at PsCreateSystemThread.

The kernel environment is just so different. A user-mode process can be careless, and all of its detritus will be cleaned up when the process ends or explodes. There is no equivalent concept in the kernel. There’s nobody to clean up after sloppy code, because a kernel process never ends. You just have to be more careful.

@Tim_Roberts said:

The versatility of the interface? Implementation of the C standard.

_beginthreadex is itself a Microsoft extension. The C standard library has no notion of threads, processes or even directories. And if you believe that _beginthreadex is more versatile than PsCreateSystemThread, then you have clearly never looked at PsCreateSystemThread.

The kernel environment is just so different. A user-mode process can be careless, and all of its detritus will be cleaned up when the process ends or explodes. There is no equivalent concept in the kernel. There’s nobody to clean up after sloppy code, because a kernel process never ends. You just have to be more careful.

What about <threads.h> from C11 standard?
Ok, so what are km/crt and all its contents for?!

@ThatsBerkan said:
This is the kernel, not some usermode application that any developer with half an IQ straight out of college (where he learned nothing as well) can write. Get over it. The last post of Peter is a good response to your “problem” already. I’d have liked to also talk about the privilege levels, the memory range (for example accessing the shared user data region), the pain in the ass to merge every changes done to the usermode API and add its kernel counterpart, the fact that most usermode apis DEPEND on the kernel apis, etc… but I sort of don’t feel like it, considering this thread shouldn’t even exist. But coming from someone who literally asked the following question on this forum: “How many spinlocks can I create?”; Let’s say that I’m not surprised.

Ok, I understand you a super smart guy with IQ of 100500. But what do you want to say about the privilege levels in the kernel? And what does it have to do with the fact that high-level crt-functions make corresponding kernel system calls, with the COMMON interface (signature) of these functions?

so what are km/crt and all its contents for

That’s a good question, actually. Somebody went to a lot of effort to create, for example, a version of “malloc.h” that’s clearly distinct from the user-mode version. And the code in it is very different…

I have no idea why.

Peter

1 Like

@“Peter_Viscarola_(OSR)” said:

The versatility of the interface? Implementation of the C standard? The ability to smoothly change the implementation? Simplifying the introduction of modern programming ideas in the kernel?

Programming in user-mode is not the same as programming in kernel-mode. And, while they share many of the same concerns, they do not share others. Like… threads. And… time. The issues you need to deal with in user-mode and in kernel-mode are very different.

Did you not look at the differences between _beginthreadex and PsCreateSystemThread? _beginthreadex was designed for user-mode use. It meets user-mode needs, and provides control for user-mode types of concerns. Likewise PsCreateSystemThread for kernel-mode. It’s a matter of the right tool for the right job. In this particular case, it’s not a matter of “standards” or “modern”… it’s a matter of suitability for purpose. I mean… you might just as well ask why _beginthreadex doesn’t allow the option to provide a handle to a process into which to create the thread or an access mask or an OBJECT_ATTRIBUTES pointer. It’s because that’s not the point of the function. But these things are vital when you’re running in kernel-mode, where the context in which you’re running may be specific or arbitrary.

You might as well, foolishly, ask why you can’t call malloc in kernel-mode. What could we possibly map that to that would make sense? The answer: Nothing. Because malloc makes sense in user-mode, but not in kernel-mode, where we have the concept of paged vs unpaged, executable vs nx, memory (for just a couple of tiny differences).

Having said that… if you want to call _beginthreadex in your kernel-mode code, and you can envision a set of configuration and constraint decisions that fit your specific requirements, nothing says you can’t implement your own version of _beginthreadex. I think that’d be a really bad idea… but it’d be a heck of a lot better than having some member of the kernel team make those configuration and constraint decisions for you, so that they probably wouldn’t fit your needs (or anybody else’s needs either).

The essence of engineering is creating suitable solutions that are fit for their purposes. NOT trying to force-fit the same paradigm into environments where that paradigm makes little sense.

Peter

Why was it decided to fill km/crt with duplicate titles and ads from UCRT? Why can’t I use any headers from UCRT with the kernel implementation? The only useful thing I saw in km /crt is macros for RAISE, CATCH… with /kernel flags, everything else duplicating headers and declarations from UCRT.
Perhaps in this way independence from visual studio runtime was achieved? But are there several compilers for drivers in Windows? What are the reasons for all this decision, which ultimately leads to the inability to compile STL normally and painlessly in kernel mode when building VS?

@“Peter_Viscarola_(OSR)” said:

so what are km/crt and all its contents for

That’s a good question, actually. Somebody went to a lot of effort to create, for example, a version of “malloc.h” that’s clearly distinct from the user-mode version. And the code in it is very different…

I have no idea why.

Peter

But malloc() and other function isn’t implemented in libcntpr.lib…

Right.

To quote one of my favorite people: “We’re gettin’ nowhere, fast” in this thread.

Peter

… which ultimately leads to the inability to compile STL normally and painlessly in kernel mode …

You’ve been talking about C, not C++. The ultimate problem with STL is exceptions. Exception handling is all nitty-gritty compiler-specific details. The Visual Studio compiler handles them by making fundamental assumptions about the environment (segment registers, special segment content) that do not apply in kernel mode. There is an exception-free version of STL that does work in the Windows kernel.

Peter is right. This discussion started off interesting, but is now utterly pointless. It is what it is. These decisions were made 25 years ago, and they aren’t going to change. Ever. When you build your operating system, please feel free to implement the full C++ standard library for drivers.

1 Like

More than 25 years ago surely. 25 years ago is when it go to release, so the engineering decisions had to have been made before then :wink:

But that’s kind of the point of the responses in this thread: Kernel programming is engineering.

As an analogy, you would not expect a bridge engineered to span Ray’s creek at Augusta national golf course to be something that you can directly reuse without modifications to span the Hudson river to Manhattan island. And you can’t reverse them either. The Brooklyn bridge wouldn’t work for Ray’s creek.

engineering requires an understanding of the problem space and the available tools. And is all about applying tools (including the human mind) towards solving problems. It is not about abstract principals but their concrete application.

Programming as engineering is not confined to KM programming on windows. Essentially everything that can be called systems programming is engineering. That includes the software than runs your car’s transmission and ABS. It includes the stuff they send to Mars or the Moon. Flight controls on an aircraft as well as the traffic lights down the street. And even an enormous array of UM programs for windows are all the products of engineering.

But to achieve concrete results, sometimes compromises have to be made. If I setup a quick poll and asked the members here, who has decided that at least in one case, copy and pasting identical code to more than one place was the best option, I expect that 99% would say yes - at least once it has been the best option. And I further expect that the other 1% simply haven’t got there yet. And that’s despite the fact that copy / paste code is amongst the worst things that a program can do. The code size will be larger and maintenance issues abound. But it is about judgement. And let’s not even start on the less equivocal issues of the use of goto or multiple exits per function; much less the ‘correct’ way to handle errors.

rather than worrying about CRT or STL, it would be MUCH more interesting to aim to adapt C# type safety with interfaces, templates and generics in KM. Much as it has been discussed, it seems an equally dim prospect however

1 Like