Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Sept/Oct 2019 Issue of The NT Insider available


Download PDF here: http://insider.osr.com/2019/ntinsider_2019_01.pdf

It’s a particularly BIG issue, too: 40 pages of technical goodness, ranging from WDF to Minifilters. Check it out.
Before Posting...
Please check out the Community Guidelines in the Announcements and Administration Category.

Alternative to PsCreateSystemThreads / Threads in total

fredantomfredantom Member Posts: 8
edited September 25 in NTDEV

Hello,

I have been given a assignment where i have to write functionality like a thread, but without actually using a thread.

So I would like to ask if there is a way to run multiple pieces of code simultaneously without using a thread (kmdf driver).

Comments

  • Scott_Noone_(OSR)Scott_Noone_(OSR) Administrator Posts: 3,172

    Why is a thread not appropriate? What are you trying to accomplish?

    -scott
    OSR

  • Tim_RobertsTim_Roberts Member - All Emails Posts: 13,102

    No. There's no way. In order to have code running on multiple CPUs at the same time, the operating system scheduler has to be involved, and the scheduler only works with threads.

    Now, you can SIMULATE threads using co-routines, but in that case the pieces of code don't actually run simultaneously. It's cooperative sharing of one single CPU.

    Tim Roberts, [email protected]
    Providenza & Boekelheide, Inc.

  • Martin_DrábMartin_Dráb Member - All Emails Posts: 66

    So I would like to ask if there is a way to run multiple pieces of code simultaneously without using a thread (kmdf driver).

    Well, you may take advantage of DPCs (with their strict limitations, e.g. DISPATCH_LEVEL IRQL, timeout limits, ...) or workitems (system worker threads are used to execute them, so you do not need to create your own threads).

    Martin Dráb

  • fredantomfredantom Member Posts: 8
    edited September 26

    @Scott_Noone_(OSR) said:
    Why is a thread not appropriate? What are you trying to accomplish?

    It's a coding "challenge" at my school, and I was wondering if anyone had a solution to it.

    @Martin_Dráb said:

    So I would like to ask if there is a way to run multiple pieces of code simultaneously without using a thread (kmdf driver).

    Well, you may take advantage of DPCs (with their strict limitations, e.g. DISPATCH_LEVEL IRQL, timeout limits, ...) or workitems (system worker threads are used to execute them, so you do not need to create your own threads).

    I was looking in to "ExQueueWorkItem" before asking this question, but I couldn't figure out if I would actually be able to run multiple instances of code simultaneously or if its all just put into a queue and executed one after another in that queue.

    I also couldn't find any examples of "usage" so I had a hard time implementing this.

    What i tried:
    PWORK_QUEUE_ITEM WorkItem = CONTAINING_RECORD(dostuff, WORK_QUEUE_ITEM, List);
    ExQueueWorkItem(WorkItem, CriticalWorkQueue);

    But I can't seem to start multiple work items at once (Bug Check 0xE4: WORKER_INVALID).

    Any help would be appreciated.

    Post edited by fredantom on
  • anton_bassovanton_bassov Member Posts: 5,052

    It's a coding "challenge" at my school, and I was wondering if anyone had a solution to it.

    ....

    I was looking in to "ExQueueWorkItem"

    I guess in such case what they expect from you is writing your own mini-scheduler. Although queuing a workitem seems to be the easiest option in existence, I seriously doubt that your mentors are going to get satisfied with such a "solution" - after all, it seems just a way to easy, don't you think.....

    Anton Bassov

  • anton_bassovanton_bassov Member Posts: 5,052

    No. There's no way. In order to have code running on multiple CPUs at the same time, the operating system scheduler
    has to be involved, and the scheduler only works with threads.

    Actually, you can.....

    What you have to do in order to achieve such a goal is to create multiple per-CPU threads - you can do it by setting the thread affinity mask, effectively binding a thread to a given CPU.

    Each of these threads is going to act as both scheduler ( i.e."mini-scheduler" that is responsible for scheduling multiple "mini-threads") , and a unit of execution ( i.e.a "mini-thread"). All this "scheduling is going to be totally completely transparent to the system scheduler. In other words, the whole thing will work pretty much like the userland thread scheduling in ancient Solaris versions.

    Once these per-CP threads may run simultaneously on different CPUs you may end up with running multiple"mini-threads" simultaneously
    if you do it this way. Therefore, you will have to implement your own synch constructs in order to make it safe.

    I think this is more or less what the OP is expected to do here.....

    Anton Bassov

  • Tim_RobertsTim_Roberts Member - All Emails Posts: 13,102

    ... what you have to do in order to achieve such a goal is to create multiple per-CPU threads.

    Maybe you didn't see his original question, where he said he couldn't use threads.

    Tim Roberts, [email protected]
    Providenza & Boekelheide, Inc.

  • Tim_RobertsTim_Roberts Member - All Emails Posts: 13,102

    I was looking in to "ExQueueWorkItem" before asking this question, but I couldn't figure out if I would actually be able to run multiple instances of code simultaneously or if its all just put into a queue and executed one after another in that queue.

    The documentation only says that system worker threads are a "limited resource". As far as I know, there is more than one, but there are no guarantees.

    ExQueueWorkItem(WorkItem, CriticalWorkQueue);
    But I can't seem to start multiple work items at once (Bug Check 0xE4: WORKER_INVALID).

    There's no need to use CriticalWorkQueue. I suspect there are more "normal" worker threads than "critical" worker threads.

    Remember that each work item has to have its own WORK_QUEUE_ITEM structure. You can't queue two work items with the same WORK_QUEUE_ITEM.

    Tim Roberts, [email protected]
    Providenza & Boekelheide, Inc.

  • anton_bassovanton_bassov Member Posts: 5,052

    Maybe you didn't see his original question, where he said he couldn't use threads.

    Well, the way I understood the OP's posts, he is just required to do something more advanced than simply creating multiple threads and letting them run. It does not necessarily imply that he is not allowed to call PsCreateSystemThread() - instead, he seems to be requested just to implement some extra scheduler-like functionality on top of it

    Anton Bassov

  • fredantomfredantom Member Posts: 8
    edited September 27

    @anton_bassov said:
    I think this is more or less what the OP is expected to do here.....
    Anton Bassov

    Is there any examples for something likes around the forum or maybe on github?> @anton_bassov said:

    Maybe you didn't see his original question, where he said he couldn't use threads.

    Well, the way I understood the OP's posts, he is just required to do something more advanced than simply creating multiple threads and letting them run. It does not necessarily imply that he is not allowed to call PsCreateSystemThread() - instead, he seems to be requested just to implement some extra scheduler-like functionality on top of it

    Anton Bassov

    I'm not allowed to use PsCreateSystemThread or any other method of directly creating a thread that could be seen as increasing the thread count of the driver.

    @Tim_Roberts said:

    I was looking in to "ExQueueWorkItem" before asking this question, but I couldn't figure out if I would actually be able to run multiple instances of code simultaneously or if its all just put into a queue and executed one after another in that queue.

    The documentation only says that system worker threads are a "limited resource". As far as I know, there is more than one, but there are no guarantees.

    ExQueueWorkItem(WorkItem, CriticalWorkQueue);
    But I can't seem to start multiple work items at once (Bug Check 0xE4: WORKER_INVALID).

    There's no need to use CriticalWorkQueue. I suspect there are more "normal" worker threads than "critical" worker threads.

    Remember that each work item has to have its own WORK_QUEUE_ITEM structure. You can't queue two work items with the same WORK_QUEUE_ITEM.

    I only made one work item and initialized it, also tried with normal queue priory but yielding same result.

    Is there any way to use std::future (since C++11) in a kmdf driver?

  • fredantomfredantom Member Posts: 8

    @anton_bassov said:

    Maybe you didn't see his original question, where he said he couldn't use threads.

    Well, the way I understood the OP's posts, he is just required to do something more advanced than simply creating multiple threads and letting them run. It does not necessarily imply that he is not allowed to call PsCreateSystemThread() - instead, he seems to be requested just to implement some extra scheduler-like functionality on top of it

    Anton Bassov

    I'm not allowed to use Threads in general, we have to implement it in another way than just creating threads using PsCreateSystemThread and calling it a day.

    @anton_bassov said:

    No. There's no way. In order to have code running on multiple CPUs at the same time, the operating system scheduler
    has to be involved, and the scheduler only works with threads.

    Actually, you can.....

    What you have to do in order to achieve such a goal is to create multiple per-CPU threads - you can do it by setting the thread affinity mask, effectively binding a thread to a given CPU.

    Each of these threads is going to act as both scheduler ( i.e."mini-scheduler" that is responsible for scheduling multiple "mini-threads") , and a unit of execution ( i.e.a "mini-thread"). All this "scheduling is going to be totally completely transparent to the system scheduler. In other words, the whole thing will work pretty much like the userland thread scheduling in ancient Solaris versions.

    Once these per-CP threads may run simultaneously on different CPUs you may end up with running multiple"mini-threads" simultaneously
    if you do it this way. Therefore, you will have to implement your own synch constructs in order to make it safe.

    I think this is more or less what the OP is expected to do here.....

    Anton Bassov

    Is there any examples on this either on the forum or on GitHub?

  • Tim_RobertsTim_Roberts Member - All Emails Posts: 13,102

    Is there any examples on this either on the forum or on GitHub?

    No, because it's silly. In the real world, people use threads. They are easy, reliable, and lightweight (on Windows). What you're exploring is a pointless academic exercise.

    If you want to do some reading, however, what Anton's talking about is the concept user-mode Windows calls "fibers". Go look up the CreateFiber function. The C++20 standard also adds co-routines to the standard library.

    However, all of these things involve threads. If you are allowed to use threads but just can't use PsCreateSystemThread, there are ways to do that. But if you really can't exploit threads, then your problem cannot be solved.

    Tim Roberts, [email protected]
    Providenza & Boekelheide, Inc.

  • anton_bassovanton_bassov Member Posts: 5,052

    I'm not allowed to use Threads in general, we have to implement it in another way than just creating threads using
    PsCreateSystemThread and calling it a day.

    Well, but you have to get to the CPU somehow,don't you think......

    Is there any examples on this either on the forum or on GitHub?

    As Tim pointed out already, from the practical standpoint the whole thing is simply stupid, particularly in its KM implementation.
    Older Solaris (and Linux as well) versions used to do it in the userland, but the whole thing was eventually abandoned in favour of delegating the task to the system scheduler, again, because it proved to be simply unreasonable. Therefore, you are very unlikely to find an implementation of it.

    BTW, you seem to be a pretty poorly-motivated student. If I was in your place I would get excited by such an assignment, because it is meant to give you the basics of the practical OS design. However, you seem to be looking for a ready solution instead....

    Anton Bassov

  • anton_bassovanton_bassov Member Posts: 5,052

    what Anton's talking about is the concept user-mode Windows calls "fibers". Go look up the CreateFiber function.

    I was under the impression that fibers had been deprecated ages ago.Am I wrong here?

    Anton Bassov

  • Tim_RobertsTim_Roberts Member - All Emails Posts: 13,102

    Many things that are "deprecated" continue to work fabulously well, and will do so into the infinite future. AHEM Video for Windows. COUGH waveIn/waveOut. DirectShow.

    Tim Roberts, [email protected]
    Providenza & Boekelheide, Inc.

  • anton_bassovanton_bassov Member Posts: 5,052

    Many things that are "deprecated" continue to work fabulously well, and will do so into the infinite future

    Fair enough....

    The "only" problem with fibers here is that the OP wants to do it in the kernel, rather than in the userland, so that he is still completely on his own with his assignment.....

    Another problem for the OP is that fibers are a way too simplistic and look more of co-routines, rather than a light-weight multithreading.
    For all practical purposes they are just cooperatively scheduled threads. If ANY of them returns, the whole program terminates, and I am not 100% sure this is what the OP's "mentors" expect to happen in the kernel....

    Anton Bassov

  • Scott_Noone_(OSR)Scott_Noone_(OSR) Administrator Posts: 3,172

    The only problem I see is that the OP asked a very specific question that is (apparently) part of a larger thought experiment to a professional developer list without stating that it was a thought experiment. This then causes said professional developers to wonder why the hell you would try to solve a problem without using the tools available.

    I’m all for these types of questions in the right environment (buy me a beer and I’ll make up anything). But here, not so much...

    -scott
    OSR

Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Upcoming OSR Seminars
Writing WDF Drivers 21 Oct 2019 OSR Seminar Space & ONLINE
Internals & Software Drivers 18 Nov 2019 Dulles, VA
Kernel Debugging 30 Mar 2020 OSR Seminar Space
Developing Minifilters 27 Apr 2020 OSR Seminar Space & ONLINE