Hi all,
This thread is the continuation of discussion from my previous thread Virtual DOS machine.
As Tim Roberts pointed out the proof of existence - Microsoft Singularity project - where all user and kernel components are JITed before execution.
I would like to develop a single driver in existing windows which does this.
With what considerations should I start off?
Singularity was an entire os dedicated to running c# at all layers of the os, it is not an existence proof that means you can bolt the same into the existing Windows driver and app model
Sent from Outlook Mailhttp: for Windows 10
From: xxxxx@gmail.com
Sent: Thursday, August 13, 2015 12:40 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Can we develop a virtual machine framework hosted within a windows kernel driver?
Hi all,
This thread is the continuation of discussion from my previous thread Virtual DOS machine.
As Tim Roberts pointed out the proof of existence - Microsoft Singularity project - where all user and kernel components are JITed before execution.
I would like to develop a single driver in existing windows which does this.
—
NTDEV is sponsored by OSR
Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</http:>
@Doron: THIS IS AWESOME. Thank you.
xxxxx@gmail.com wrote:
This thread is the continuation of discussion from my previous thread Virtual DOS machine.
As Tim Roberts pointed out the proof of existence - Microsoft Singularity project - where all user and kernel components are JITed before execution.
I would like to develop a single driver in existing windows which does this.
The first thing you need to do is sit down with a blank sheet of paper
and do some high-level design work to figure out what it is you’re going
to do. Right now, you have an overly generic task description – one
that is so generic it cannot be implemented. Draw some flow charts –
what goes in, what comes out.
Let’s start with the byte code. How are you going to generate that? It
would be silly for you to create your own intermediate language and
write your own compiler. That’s a huge problem has already been solved
multiple times. In what language do you want to write your managed
drivers? There are about 40 languages that can produce Microsoft’s .NET
IL. There are about 30 languages that can produce Java VM IL. I’m sure
there are others. You could even do a Forth environment.
Then, you have to figure out how you are going to handle the
management. In a .NET application, there is a stub “main” function in
native code that loads and transfers control to the .NET runtime. It
then takes over and runs the app. You could do that. Or, for a proof
of concept, you could have the driver be the runtime, and have it load
the IL separately.
After that, you have to figure out how to provide kernel services to
your IL. What abstractions will you provide? How will you handle
dispatching? Are you going to duplicate what’s already in KMDF and just
provide simple transitions back and forth?
You have a long way to go before you can even know how big this project is.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
Let’s back up a second. I’m not sure what we’re discussing.
Mr Kumbhar’s initial question was:
I *think* what Mr Kumbhar means is “Can a Windows driver implement an interpreter and host interpreted code.” If that’s wrong, Mr Kumbhar, please let me know.
Assuming my re-read of his question is correct, the answer is “of course, obviously.” In terms of how to design it, you write some code that interprets byte-code. Your driver opens a file, reads it in, and executes it. Done.
Whether the MSR OS Singularity ever existing is entirely irrelevant to this question.
There is at least one driver in everyone’s system today that IS in fact an interpreter: It’s ACPI.
So… what’s the question again?
Peter
OSR
@OSRDrivers
@Tim: Otherwise, I had started of by downloading MS Singularity and reading its introductory docs. And for accessing kernel services I had thought about creating wrappers. As for bytecode, I had listed out not only MSIL but other virtual machine intermediate language as well. For example, Forth, Neko etc. As far compiler is considered still searching for an open source one which will generate bytecode.
That’s ok.
@Peter: You are correct. That is it. Exactly. So if I understand ACPI driver development, I can reuse the concept. Ok. Then this is not a huge project. It can be enclosed within solid boundaries of the WDK. I am very happy to know this. But again my idea is to host a general purpose framework. So I consider what Tim says - Its a long way ahead.
Before moving ahead with the project, I will go through ACPI as Peter has pointed out. That seems extremely feasible at this point since I am still analyzing how to proceed.
I don’t know what this means, and unless you have access to Windows source code, you’re not going to be able to “go through” (in the sense of “refer to”, “review” or use as a reference) the ACPI driver.
My point is that your project is demonstrably possible, without all the hoopla associated with JIT’ing or anything else. And it’s also absolutely trivial and without any special challenge. Get a copy of an open source BASIC interpreter written in C and stuff it into your driver (here’s one that might work from Github: http://github.com/adamdunkels/ubasic). And… OK. Now you have a BASIC interpreter running in your driver. Yay! Or, more likely, Yawn!
The larger point to consider is “What do you gain by doing this?” Seriously. Isn’t it a big “who cares?”
Peter
OSR
@OSRDrivers
xxxxx@gmail.com wrote:
@Peter: You are correct. That is it. Exactly. So if I understand ACPI driver development, I can reuse the concept. Ok. Then this is not a huge project. It can be enclosed within solid boundaries of the WDK. I am very happy to know this. But again my idea is to host a general purpose framework. So I consider what Tim says - Its a long way ahead.
Before moving ahead with the project, I will go through ACPI as Peter has pointed out. That seems extremely feasible at this point since I am still analyzing how to proceed.
Right, it depends on what you are hoping to do. ACPI has defined its
own language (ACPI Source Language, or ASL) that describes the resources
a BIOS needs to provide, and methods for accessing those resources. You
can see an example here:
http://stackoverflow.com/questions/4314407/help-me-decipher-this-acpi-source-language-code
If you were trying to provide some master driver for a group of related
devices, where they share many of the same operations with differences
in detail, then it makes good sense to invent that kind of a language,
so that your master driver can interpret the compiled result to provide
a specific device.
But if you are trying to construct something that can be used to write
arbitrary drivers, then you’ve moved up to the Singularity level.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
To be clear… Don’t head off to download and install Singularity just so you can review the imagined wonders of it’s driver model. Because it doesn’t HAVE a driver model.
I only looked at Singularity briefly (I spent almost two years working on Midori, which was based on the early work done on Singularity), but IIRC Singularity doesn’t HAVE a driver model, it doesn’t HAVE an I/O subsystem and it supports about 5 devices that are integrated into the system using custom written code that’s all treated as a special case.
The point of Singularity was to study and prototype issues concerning code that’s “provably correct” by construction and Software Isolated Processes. It served this purpose very well, and the OS concepts it demonstrates were, to many including me, super inspiring. I know some of the researchers who worked on Singularity, and to say that I’m awed by the stuff they think of and work on daily is a major understatement.
The guys who study these things aren’t much interested in mundane topics like I/O subsystems, however. They typically view drivers as troublesome and uninteresting necessities, akin to the command shell, and as such these annoyances are paid little attention, created with little thought, and dealt with later as little as possible. The goal of drivers in a research operating system like Singularity is to support a disk, a video display, a keyboard and maybe a mouse and a network card and be done with it so the project can move on to doing interesting and more important things.
Thus, the moral of the story is: If you’re interested in I/O Subsystems, then looking at the Singularity OS won’t provide any information, data, or model that’ll be directly applicable. If you’re interested in the overall concept of provably correct code and/or software isolation among processes, then Singularity will be very instructive.
Like I said, I don’t want our previous posts in this and the other thread to send somebody off to waste their timing digging into Singularity thinking “Oh, cool… I can see how interpreted code can be used to create a general purpose I/O subsystem and driver model” (or something) because if they DO take the time, I think they’ll be pretty underwhelmed by what they find.
OTOH, I’m waiting for Microsoft to put the Midori sources on GitHub – and before you get too excited, I don’t have any inside info on this and I’m quite sure I’ll be waiting an infinite amount of time for it to happen… I’m just saying it would be very cool).
Peter
OSR
@OSRDrivers
I dont mean Windows source code literally. What I meant to say is,
Microsoft’s documentation about ACPI does mention AML Language of ACPI. So
I will learn a bit about that and probably develop an dummy ACPI driver for
learning. To understand the concept.
“EASY MAINTENANCE” is the primary thing I gain.
For eg. Say I have a filter driver who registers for flt_file_create event
only and monitors file creation.
Next I want to monitor changes to file. So I would subscribe for
flt_file_write event. This will cause me to make changes to source code and
recompile the driver. ***I do not want to recompile the driver. I only want
to compile the code that registers for flt_file_write and its associated
logic.*** Currently I cannot do this. Which is what I want to achieve.
Its about accessing kernel services from byte-code which is ***at this
point of time*** a special challenge and not trivial ***for me***.
No. No yawning. Instead I could use this as a hands on and a starting point
for what I am looking.
On Sat, Aug 15, 2015 at 7:28 PM, wrote:
>
>
> To be clear… Don’t head off to download and install Singularity just so
> you can review the imagined wonders of it’s driver model. Because it
> doesn’t HAVE a driver model.
>
>
>
> I only looked at Singularity briefly (I spent almost two years working on
> Midori, which was based on the early work done on Singularity), but IIRC
> Singularity doesn’t HAVE a driver model, it doesn’t HAVE an I/O subsystem
> and it supports about 5 devices that are integrated into the system using
> custom written code that’s all treated as a special case.
>
> The point of Singularity was to study and prototype issues concerning code
> that’s “provably correct” by construction and Software Isolated Processes.
> It served this purpose very well, and the OS concepts it demonstrates were,
> to many including me, super inspiring. I know some of the researchers who
> worked on Singularity, and to say that I’m awed by the stuff they think of
> and work on daily is a major understatement.
>
> The guys who study these things aren’t much interested in mundane topics
> like I/O subsystems, however. They typically view drivers as troublesome
> and uninteresting necessities, akin to the command shell, and as such these
> annoyances are paid little attention, created with little thought, and
> dealt with later as little as possible. The goal of drivers in a research
> operating system like Singularity is to support a disk, a video display, a
> keyboard and maybe a mouse and a network card and be done with it so the
> project can move on to doing interesting and more important things.
>
> Thus, the moral of the story is: If you’re interested in I/O Subsystems,
> then looking at the Singularity OS won’t provide any information, data, or
> model that’ll be directly applicable. If you’re interested in the overall
> concept of provably correct code and/or software isolation among processes,
> then Singularity will be very instructive.
>
>
>
> Like I said, I don’t want our previous posts in this and the other thread
> to send somebody off to waste their timing digging into Singularity
> thinking “Oh, cool… I can see how interpreted code can be used to create
> a general purpose I/O subsystem and driver model” (or something) because if
> they DO take the time, I think they’ll be pretty underwhelmed by what they
> find.
>
> OTOH, I’m waiting for Microsoft to put the Midori sources on GitHub – and
> before you get too excited, I don’t have any inside info on this and I’m
> quite sure I’ll be waiting an infinite amount of time for it to happen…
> I’m just saying it would be very cool).
>
> Peter
> OSR
> @OSRDrivers
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
But… but… recompiling the driver takes 20 seconds! Designing, coding and testing the change you want to implement takes many hours. What’s the win?
But… but… accessing kernel services from byte-code means accessing your interpreter’s machine language code that calls the kernel services from machine language. So… all you get to prove is that you can have a sequence of byte code pass parameters to your own purpose-built interface that grabs those parameters and in turn calls the kernel service.
If you’ve never written an interpreter before, I *guess* that’s *sort* of interesting… but you’d be far better off implementing a byte-code interpreter that can do printf in user-mode if your goal is to simply understand interpreters.
Sorry, I don’t understand at all what you’re saving or gaining in any of this.
Nonetheless, I wish you good luck in your project.
Peter
OSR
@OSRDrivers
Basically, what the OP seems to be up to is just instroducing the LISP concepts to the kernel.
Unlike the languages like C,C++,C# et al LISP does not really make any distinction betweeen the code and data, which means LISP program may happily write its own code. More on it below…
Check the link below - unlike most other technical sources the author tries to explain these concepts in “rather philosophical” manner, starting from XML-based examples
http://www.defmacro.org/ramblings/lisp.html
Certainly, none of these concepts seem to apply in the kernel, but this is already another story…
Anton Bassov
On 15-Aug-2015 23:31, xxxxx@osr.com wrote:
But… but… recompiling the driver takes 20 seconds! Designing, coding and testing the change you want to implement takes many hours. What’s the win?
Er… perhaps, the signing? Sign once, hack anytime 
– pa
Thank you.
On Sun, Aug 16, 2015 at 8:04 AM, Pavel A. wrote:
> On 15-Aug-2015 23:31, xxxxx@osr.com wrote:
>
>> But… but… recompiling the driver takes 20 seconds! Designing, coding
>> and testing the change you want to implement takes many hours. What’s the
>> win?
>>
>>
> Er… perhaps, the signing? Sign once, hack anytime
>
> – pa
>
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
How VERY dumb of me to miss that Mr. Pavel A, thank you – Of course, that’s the point here… isn’t it Mr. Kumbhar? The point is to create a variety of loadable exploits that are interpreted by a signed driver.
Peter
OSR
@OSRDrivers
And if such a driver somehow got a production signature, Microsoft would almost certainly instantly revoke it’s signature.
For lab/testing/debugging use, I could see valid uses for a driver that you passed data to execute/interpret. A test signature might be perfectly sufficient for such a driver. Keep in mind Microsoft is now targeting Windows for the embedded/hacking world (it now runs on a Raspberry Pi 2) and solutions that I would class as downright obscene for a datacenter server might be completely appropriate for a machine running a little experiment in some college lab. A researcher who needs to control some bits on a lab device likly has about zero interest in learning how to write production Windows drivers.
Jan
On 8/16/15, 8:46 AM, “xxxxx@lists.osr.com on behalf of xxxxx@osr.com” wrote:
> The point is to create a variety of loadable exploits that are interpreted by a signed driver.
Lets Back up a bit… This discussion started off because I read about Virtual Device Drivers which I found have become obselete. Then I found currently there is Device Type - VDM (#define FILE_DEVICE_VDM 0x0000002c). Then found out about ntvdm.exe… Then for 64 bit windows ntvdm is absent ** according to previous posts of the discussion on another thread ** Then Your’s, Tim’s and Doran’s suggestions - Singularity, ACPI etc. And the discussion was knowledgeable. I just wanted to learn.
So Mr. Peter… My stand on your remark:
@JAN:
Its like building a concept car.
That’s why support for directly manipulating GPIOs and I2C controllers and such was added to C#… so it can be used on IoT Core, with no driver development needed.
Peter
OSR
@OSRDrivers