Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Home NTDEV

More Info on Driver Writing and Debugging


The free OSR Learning Library has more than 50 articles on a wide variety of topics about writing and debugging device drivers and Minifilters. From introductory level to advanced. All the articles have been recently reviewed and updated, and are written using the clear and definitive style you've come to expect from OSR over the years.


Check out The OSR Learning Library at: https://www.osr.com/osr-learning-library/


Before Posting...

Please check out the Community Guidelines in the Announcements and Administration Category.

About UCRT and KM/CRT

Alex_FunkyAlex_Funky Member - All Emails Posts: 186

Oh, that's pretty hard... I'm trying to understand how the C standard is implemented in the kernel, and what justifies the existence of km/crt in the wdk, why can't I use headers from ucrt in the kernel while doing the implementation for the kernel? All this 'defs hell' and the fact that libcntpr.lib does not implement everything that is declared in the headers (malloc, open, clock, etc)... Why can't I make a ucrt with a kernel and user implementation? Or am I just missing something?

Comments

  • MBond2MBond2 Member Posts: 304

    well, if you want to know all of that, you better be prepared to fork over a bottle of scotch or two

    remember that the C language is a standard for converting 'high level' code into machine language on a specific platform. Here we deal mostly with x64, x86, ARM and that ITANIUM thingy. but the C language has been ported to hundreds of other platforms including the ones that run the stop lights in you neighborhood most likely

    but of course the CRT is another matter. Programs that tell you to walk or don't walk, usually don't need access to cosines, tangents or psudo-random numbers. They also don't really need dynamic memory allocation either. The same is true of the programs that control the breaks on your car and the ventilators in the hospital.

    KM programs in Windows also have to deal with a subset - although not nearly as restrictive as in some other environments

  • Tim_RobertsTim_Roberts Member - All Emails Posts: 13,909

    Why can't I make a ucrt with a kernel and user implementation?

    That is EXACTLY what libcntpr.lib is -- the kernel implementation of ucrt. The things that aren't implemented are things that need more information (malloc -- you need to specify what pool to use), things that do not allow for appropriate error handling (open/fopen and friend), and other things that a kernel driver shouldn't be doing anyway. A kernel driver is supposed to be little more than glue. You shouldn't be writing applications in kernel mode.

    Tim Roberts, [email protected]
    Providenza & Boekelheide, Inc.

  • Alex_FunkyAlex_Funky Member - All Emails Posts: 186

    @Tim_Roberts said:

    Why can't I make a ucrt with a kernel and user implementation?

    That is EXACTLY what libcntpr.lib is -- the kernel implementation of ucrt. The things that aren't implemented are things that need more information (malloc -- you need to specify what pool to use), things that do not allow for appropriate error handling (open/fopen and friend), and other things that a kernel driver shouldn't be doing anyway. A kernel driver is supposed to be little more than glue. You shouldn't be writing applications in kernel mode.

    what about _beginthreadex(), what about clock() and others - what are the fundamental problems to implement them?

  • Alex_FunkyAlex_Funky Member - All Emails Posts: 186

    @MBond2 said:
    well, if you want to know all of that, you better be prepared to fork over a bottle of scotch or two

    remember that the C language is a standard for converting 'high level' code into machine language on a specific platform. Here we deal mostly with x64, x86, ARM and that ITANIUM thingy. but the C language has been ported to hundreds of other platforms including the ones that run the stop lights in you neighborhood most likely

    but of course the CRT is another matter. Programs that tell you to walk or don't walk, usually don't need access to cosines, tangents or psudo-random numbers. They also don't really need dynamic memory allocation either. The same is true of the programs that control the breaks on your car and the ventilators in the hospital.

    KM programs in Windows also have to deal with a subset - although not nearly as restrictive as in some other environments

    ok, but what is the reason that many headers are duplicated in km/crt from ucrt, while many are added (crtdefs.h, etc), which violate the general semantics of ucrt, which makes it impossible to use third-party implementations of some things based on standard headers from ucrt? When I connect the <time.h> I expect all functions from it to be implemented, otherwise, why are they defined there at all?

  • Tim_RobertsTim_Roberts Member - All Emails Posts: 13,909

    I'm not going to defend every choice. The NT kernel was simply not designed to run arbitrary general-purpose applications. They provided routines that are useful and safe. Kernel code needs to be extremely careful and handle error conditions properly. The CRT routines encourage you to forget about all of that.

    Tim Roberts, [email protected]
    Providenza & Boekelheide, Inc.

  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 8,399

    what are the fundamental problems to implement them?

    Well, there aren’t necessarily any fundamental problems implementing these... but they are user mode APIs with user mode conventions and constraints... and these have kernel mode equivalents that have their own, specific, kernel mode conventions and constraints. There’s just no reason to force-fit user mode conventions into the OS. Just compare _beginthreadex() to PsCreateSystemThread(). Why use a sledge hammer to make the former into the latter? I mean... who would want to use this?

    Peter

    Peter Viscarola
    OSR
    @OSRDrivers

  • Alex_FunkyAlex_Funky Member - All Emails Posts: 186

    @Peter_Viscarola_(OSR) said:

    what are the fundamental problems to implement them?

    Well, there aren’t necessarily any fundamental problems implementing these... but they are user mode APIs with user mode conventions and constraints... and these have kernel mode equivalents that have their own, specific, kernel mode conventions and constraints. There’s just no reason to force-fit user mode conventions into the OS. Just compare _beginthreadex() to PsCreateSystemThread(). Why use a sledge hammer to make the former into the latter? I mean... who would want to use this?

    Peter

    The versatility of the interface? Implementation of the C standard? The ability to smoothly change the implementation? Simplifying the introduction of modern programming ideas in the kernel?

  • Alex_FunkyAlex_Funky Member - All Emails Posts: 186

    Вопрос был в другом :> @Tim_Roberts said:

    I'm not going to defend every choice. The NT kernel was simply not designed to run arbitrary general-purpose applications. They provided routines that are useful and safe. Kernel code needs to be extremely careful and handle error conditions properly. The CRT routines encourage you to forget about all of that.

    The question was asked in general : why are headers and definitions duplicated in the UCRT folder and KM/CRT folder and the same functionality is not provided? How do I find out what exactly is implemented from the malloc.h, stdlib.h, etc. headers in libcntpr.lib, and what is not? Then why are there all these headers and definitions that I can connect to the project, but I can't use them?

  • ThatsBerkanThatsBerkan Member Posts: 56
    edited January 15

    This is the kernel, not some usermode application that any developer with half an IQ straight out of college (where he learned nothing as well) can write. Get over it. The last post of Peter is a good response to your "problem" already. I'd have liked to also talk about the privilege levels, the memory range (for example accessing the shared user data region), the pain in the ass to merge every changes done to the usermode API and add its kernel counterpart, the fact that most usermode apis DEPEND on the kernel apis, etc... but I sort of don't feel like it, considering this thread shouldn't even exist. But coming from someone who literally asked the following question on this forum: "How many spinlocks can I create?"; Let's say that I'm not surprised.

  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 8,399
    edited January 15

    The versatility of the interface? Implementation of the C standard? The ability to smoothly change the implementation? Simplifying the introduction of modern programming ideas in the kernel?

    Programming in user-mode is not the same as programming in kernel-mode. And, while they share many of the same concerns, they do not share others. Like... threads. And... time. The issues you need to deal with in user-mode and in kernel-mode are very different.

    Did you not look at the differences between _beginthreadex and PsCreateSystemThread? _beginthreadex was designed for user-mode use. It meets user-mode needs, and provides control for user-mode types of concerns. Likewise PsCreateSystemThread for kernel-mode. It's a matter of the right tool for the right job. In this particular case, it's not a matter of "standards" or "modern"... it's a matter of suitability for purpose. I mean... you might just as well ask why _beginthreadex doesn't allow the option to provide a handle to a process into which to create the thread or an access mask or an OBJECT_ATTRIBUTES pointer. It's because that's not the point of the function. But these things are vital when you're running in kernel-mode, where the context in which you're running may be specific or arbitrary.

    You might as well, foolishly, ask why you can't call malloc in kernel-mode. What could we possibly map that to that would make sense? The answer: Nothing. Because malloc makes sense in user-mode, but not in kernel-mode, where we have the concept of paged vs unpaged, executable vs nx, memory (for just a couple of tiny differences).

    Having said that... if you want to call _beginthreadex in your kernel-mode code, and you can envision a set of configuration and constraint decisions that fit your specific requirements, nothing says you can't implement your own version of _beginthreadex. I think that'd be a really bad idea... but it'd be a heck of a lot better than having some member of the kernel team make those configuration and constraint decisions for you, so that they probably wouldn't fit your needs (or anybody else's needs either).

    The essence of engineering is creating suitable solutions that are fit for their purposes. NOT trying to force-fit the same paradigm into environments where that paradigm makes little sense.

    Peter

    Peter Viscarola
    OSR
    @OSRDrivers

  • Tim_RobertsTim_Roberts Member - All Emails Posts: 13,909

    The versatility of the interface? Implementation of the C standard.

    _beginthreadex is itself a Microsoft extension. The C standard library has no notion of threads, processes or even directories. And if you believe that _beginthreadex is more versatile than PsCreateSystemThread, then you have clearly never looked at PsCreateSystemThread.

    The kernel environment is just so different. A user-mode process can be careless, and all of its detritus will be cleaned up when the process ends or explodes. There is no equivalent concept in the kernel. There's nobody to clean up after sloppy code, because a kernel process never ends. You just have to be more careful.

    Tim Roberts, [email protected]
    Providenza & Boekelheide, Inc.

  • Alex_FunkyAlex_Funky Member - All Emails Posts: 186

    @Tim_Roberts said:

    The versatility of the interface? Implementation of the C standard.

    _beginthreadex is itself a Microsoft extension. The C standard library has no notion of threads, processes or even directories. And if you believe that _beginthreadex is more versatile than PsCreateSystemThread, then you have clearly never looked at PsCreateSystemThread.

    The kernel environment is just so different. A user-mode process can be careless, and all of its detritus will be cleaned up when the process ends or explodes. There is no equivalent concept in the kernel. There's nobody to clean up after sloppy code, because a kernel process never ends. You just have to be more careful.

    What about <threads.h> from C11 standard?
    Ok, so what are km/crt and all its contents for?!

  • Alex_FunkyAlex_Funky Member - All Emails Posts: 186

    @ThatsBerkan said:
    This is the kernel, not some usermode application that any developer with half an IQ straight out of college (where he learned nothing as well) can write. Get over it. The last post of Peter is a good response to your "problem" already. I'd have liked to also talk about the privilege levels, the memory range (for example accessing the shared user data region), the pain in the ass to merge every changes done to the usermode API and add its kernel counterpart, the fact that most usermode apis DEPEND on the kernel apis, etc... but I sort of don't feel like it, considering this thread shouldn't even exist. But coming from someone who literally asked the following question on this forum: "How many spinlocks can I create?"; Let's say that I'm not surprised.

    Ok, I understand you a super smart guy with IQ of 100500. But what do you want to say about the privilege levels in the kernel? And what does it have to do with the fact that high-level crt-functions make corresponding kernel system calls, with the COMMON interface (signature) of these functions?

  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 8,399

    so what are km/crt and all its contents for

    That's a good question, actually. Somebody went to a lot of effort to create, for example, a version of "malloc.h" that's clearly distinct from the user-mode version. And the code in it is very different...

    I have no idea why.

    Peter

    Peter Viscarola
    OSR
    @OSRDrivers

  • Alex_FunkyAlex_Funky Member - All Emails Posts: 186

    @Peter_Viscarola_(OSR) said:

    The versatility of the interface? Implementation of the C standard? The ability to smoothly change the implementation? Simplifying the introduction of modern programming ideas in the kernel?

    Programming in user-mode is not the same as programming in kernel-mode. And, while they share many of the same concerns, they do not share others. Like... threads. And... time. The issues you need to deal with in user-mode and in kernel-mode are very different.

    Did you not look at the differences between _beginthreadex and PsCreateSystemThread? _beginthreadex was designed for user-mode use. It meets user-mode needs, and provides control for user-mode types of concerns. Likewise PsCreateSystemThread for kernel-mode. It's a matter of the right tool for the right job. In this particular case, it's not a matter of "standards" or "modern"... it's a matter of suitability for purpose. I mean... you might just as well ask why _beginthreadex doesn't allow the option to provide a handle to a process into which to create the thread or an access mask or an OBJECT_ATTRIBUTES pointer. It's because that's not the point of the function. But these things are vital when you're running in kernel-mode, where the context in which you're running may be specific or arbitrary.

    You might as well, foolishly, ask why you can't call malloc in kernel-mode. What could we possibly map that to that would make sense? The answer: Nothing. Because malloc makes sense in user-mode, but not in kernel-mode, where we have the concept of paged vs unpaged, executable vs nx, memory (for just a couple of tiny differences).

    Having said that... if you want to call _beginthreadex in your kernel-mode code, and you can envision a set of configuration and constraint decisions that fit your specific requirements, nothing says you can't implement your own version of _beginthreadex. I think that'd be a really bad idea... but it'd be a heck of a lot better than having some member of the kernel team make those configuration and constraint decisions for you, so that they probably wouldn't fit your needs (or anybody else's needs either).

    The essence of engineering is creating suitable solutions that are fit for their purposes. NOT trying to force-fit the same paradigm into environments where that paradigm makes little sense.

    Peter

    Why was it decided to fill km/crt with duplicate titles and ads from UCRT? Why can't I use any headers from UCRT with the kernel implementation? The only useful thing I saw in km /crt is macros for RAISE, CATCH... with /kernel flags, everything else duplicating headers and declarations from UCRT.
    Perhaps in this way independence from visual studio runtime was achieved? But are there several compilers for drivers in Windows? What are the reasons for all this decision, which ultimately leads to the inability to compile STL normally and painlessly in kernel mode when building VS?

  • Alex_FunkyAlex_Funky Member - All Emails Posts: 186

    @Peter_Viscarola_(OSR) said:

    so what are km/crt and all its contents for

    That's a good question, actually. Somebody went to a lot of effort to create, for example, a version of "malloc.h" that's clearly distinct from the user-mode version. And the code in it is very different...

    I have no idea why.

    Peter

    But malloc() and other function isn't implemented in libcntpr.lib......

  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 8,399

    Right.

    To quote one of my favorite people: “We’re gettin’ nowhere, fast” in this thread.

    Peter

    Peter Viscarola
    OSR
    @OSRDrivers

  • Tim_RobertsTim_Roberts Member - All Emails Posts: 13,909
    edited January 15

    ... which ultimately leads to the inability to compile STL normally and painlessly in kernel mode ...

    You've been talking about C, not C++. The ultimate problem with STL is exceptions. Exception handling is all nitty-gritty compiler-specific details. The Visual Studio compiler handles them by making fundamental assumptions about the environment (segment registers, special segment content) that do not apply in kernel mode. There is an exception-free version of STL that does work in the Windows kernel.

    Peter is right. This discussion started off interesting, but is now utterly pointless. It is what it is. These decisions were made 25 years ago, and they aren't going to change. Ever. When you build your operating system, please feel free to implement the full C++ standard library for drivers.

    Tim Roberts, [email protected]
    Providenza & Boekelheide, Inc.

  • MBond2MBond2 Member Posts: 304

    More than 25 years ago surely. 25 years ago is when it go to release, so the engineering decisions had to have been made before then ;)

    But that's kind of the point of the responses in this thread: Kernel programming is engineering.

    As an analogy, you would not expect a bridge engineered to span Ray's creek at Augusta national golf course to be something that you can directly reuse without modifications to span the Hudson river to Manhattan island. And you can't reverse them either. The Brooklyn bridge wouldn't work for Ray's creek.

    engineering requires an understanding of the problem space and the available tools. And is all about applying tools (including the human mind) towards solving problems. It is not about abstract principals but their concrete application.

    Programming as engineering is not confined to KM programming on windows. Essentially everything that can be called systems programming is engineering. That includes the software than runs your car's transmission and ABS. It includes the stuff they send to Mars or the Moon. Flight controls on an aircraft as well as the traffic lights down the street. And even an enormous array of UM programs for windows are all the products of engineering.

    But to achieve concrete results, sometimes compromises have to be made. If I setup a quick poll and asked the members here, who has decided that at least in one case, copy and pasting identical code to more than one place was the best option, I expect that 99% would say yes - at least once it has been the best option. And I further expect that the other 1% simply haven't got there yet. And that's despite the fact that copy / paste code is amongst the worst things that a program can do. The code size will be larger and maintenance issues abound. But it is about judgement. And let's not even start on the less equivocal issues of the use of goto or multiple exits per function; much less the 'correct' way to handle errors.

    rather than worrying about CRT or STL, it would be MUCH more interesting to aim to adapt C# type safety with interfaces, templates and generics in KM. Much as it has been discussed, it seems an equally dim prospect however

  • Tim_RobertsTim_Roberts Member - All Emails Posts: 13,909

    Interfaces, generics, and templates work just fine in KM. I've used them all, in many projects. (Audio and video streaming drivers have been in C++ since the 20th Century.) Just about the only major missing feature is exceptions, and that rules out STL, which is a shame.

    Tim Roberts, [email protected]
    Providenza & Boekelheide, Inc.

  • Alex_FunkyAlex_Funky Member - All Emails Posts: 186

    Thx for the discussion, guys! This is my mistake, I saw the km/crt folder and for some reason decided that this is a full-fledged implementation of C standard =(

Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Upcoming OSR Seminars
OSR has suspended in-person seminars due to the Covid-19 outbreak. But, don't miss your training! Attend via the internet instead!
Developing Minifilters 24 May 2021 Live, Online
Writing WDF Drivers 14 June 2021 Live, Online
Internals & Software Drivers 2 August 2021 Live, Online
Kernel Debugging 27 Sept 2021 Live, Online