Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results
Before Posting...
Please check out the Community Guidelines in the Announcements and Administration Category.More Info on Driver Writing and Debugging
The free OSR Learning Library has more than 50 articles on a wide variety of topics about writing and debugging device drivers and Minifilters. From introductory level to advanced. All the articles have been recently reviewed and updated, and are written using the clear and definitive style you've come to expect from OSR over the years.
Check out The OSR Learning Library at: https://www.osr.com/osr-learning-library/
Flat Executable Compiler Output
Aram_Havarneanu-2
Member Posts: 161
in NTDEV
Hello,
Is it possible to get the linker that comes with the WDK to output a flat
executable (instead of PE_COFF)?
I want to develop a bootloader for my hobby operating system. Right now, I
am booting my (PE_COFF) kernel with grub, but I wish to develop my own
bootloader. Grub does not understand PE_COFF, but it is multiboot compliant.
By carefully crafting a PE file, I can tell grub, with the aid of the
multiboot support, where to load stuff from it. This is very fragile, plus
I'd want my own bootloader anyway. The problem is that the build tools in
the WDK only generate PE_COFF files, while the bootloader needs to be a flat
binary for it to load directly into memory.
If I can't use the C compilers to get a flat file, I can write it in
assembly without any problem, but I still need linker support (which I think
it is lacking).
I could use gas (GNU) or some other tools, but what I am really trying to do
is to use the WDK build environment for everything. I don't want to mix and
maintain 2 build environments. How does the Windows team build it's
bootloader?
setenv.bat from the WDK has a HAL environment option. Is that of any use to
me?
Thanks in advance for any suggestions.
--
Aram Hăvărneanu
Is it possible to get the linker that comes with the WDK to output a flat
executable (instead of PE_COFF)?
I want to develop a bootloader for my hobby operating system. Right now, I
am booting my (PE_COFF) kernel with grub, but I wish to develop my own
bootloader. Grub does not understand PE_COFF, but it is multiboot compliant.
By carefully crafting a PE file, I can tell grub, with the aid of the
multiboot support, where to load stuff from it. This is very fragile, plus
I'd want my own bootloader anyway. The problem is that the build tools in
the WDK only generate PE_COFF files, while the bootloader needs to be a flat
binary for it to load directly into memory.
If I can't use the C compilers to get a flat file, I can write it in
assembly without any problem, but I still need linker support (which I think
it is lacking).
I could use gas (GNU) or some other tools, but what I am really trying to do
is to use the WDK build environment for everything. I don't want to mix and
maintain 2 build environments. How does the Windows team build it's
bootloader?
setenv.bat from the WDK has a HAL environment option. Is that of any use to
me?
Thanks in advance for any suggestions.
--
Aram Hăvărneanu
| Upcoming OSR Seminars | ||
|---|---|---|
| OSR has suspended in-person seminars due to the Covid-19 outbreak. But, don't miss your training! Attend via the internet instead! | ||
| Writing WDF Drivers | 7 Dec 2020 | LIVE ONLINE |
| Internals & Software Drivers | 25 Jan 2021 | LIVE ONLINE |
| Developing Minifilters | 8 March 2021 | LIVE ONLINE |
Comments
>
> Is it possible to get the linker that comes with the WDK to output a
> flat executable (instead of PE_COFF)?
No, the Microsoft 32-bit linker won't do that. There are certainly
other 32-bit linkers available that do have that ability. Check the web
pages for nasm and fasm.
The Microsoft 16-bit segmented linker can produce a flat executable (a
.com file), but you can't feed it 32-bit object files.
> I want to develop a bootloader for my hobby operating system. Right
> now, I am booting my (PE_COFF) kernel with grub, but I wish to develop
> my own bootloader. ...
> If I can't use the C compilers to get a flat file, I can write it in
> assembly without any problem, but I still need linker support (which I
> think it is lacking).
Right. The compiler is not the issue. It's the linker.
> I could use gas (GNU) or some other tools, but what I am really trying
> to do is to use the WDK build environment for everything. I don't want
> to mix and maintain 2 build environments. How does the Windows team
> build it's bootloader?
Take a look at it in a hex editor. It's a 16-bit .COM file stub (which
is flat), plus a 32-bit executable tacked on to the end. The 16-bit
stub has enough smarts to load the PE file that's tacked on the end.
--
Tim Roberts, [email protected]
Providenza & Boekelheide, Inc.
Tim Roberts, [email protected]
Providenza & Boekelheide, Inc.
I'm not sure that this will necessarily help you, but you might want to take a look at the '<ddkroot>\bin\bin16' subdirectory. In addition to the regular (link16, cl16, masm) tools, there's also exe2bin, which might be able to help out. If I recall correctly, it accepts a load address as an argument, but then again, I think I was about 14 the last and only time that I ran across it.
Now, even assuming these all work for you, exactly how one goes about using these as a part of a proper DDK BUILD environment, I haven't a clue, and as Tim already mentioned, it looks like msft rolled their own anyway. My only thoughts on this subject would be to take a look at the samples that come with this DDK involving Win95/98/Me support or VDD drivers and then take a tour of the MSDN Subscriber Downloads 'Tools and Resources' section (and probably 'Developer Tools' as well) and see what you can find in the way of old DDK's/Tools. There might be some old examples that show how to do something along these lines; not very likely, but it's the best thing I can think of. In particular, that 'Tools and Resources' section contains some things like the NT 3.51 DDK, as well as host of things that I've never even heard of.
Personally, while I think the goal of having one build environment obviously makes a lot of sense, I'd just go with gnu tools for this. In particular, yasm (what nasm is called these days, not that either is gnu, strictly speaking) could really make most of this pretty easy for you.
Good luck,
mm
> I'd just go with gnu tools for this.
Outputs of GNU tools always come with a header pre- pended . Therefore, you will have to write a helper app that strips your target of this header if you use binutils....
> In particular, yasm (what nasm is called these days, not that either is gnu, strictly speaking)
> could really make most of this pretty easy for you.
NASM is, indeed, just an excellent option for bootloader writer...
In any case, what surprises me here is the OP's choice of WDK as of his build environment for his _own_ OS - this option seems to be quite strange for someone who wants to build a custom OS (unless the target OS somehow relies upon illegitimately obtained Windows sources, of course).....
Anton Bassov
objcopy --output-target binary <in> <out>
mm
> I'm not sure that this will necessarily help you, but you might want to take a look at the '<ddkroot>\bin\bin16' subdirectory. In addition to the regular (link16, cl16, masm) tools, there's also exe2bin, which might be able to help out. If I recall correctly, it accepts a load address as an argument, but then again, I think I was about 14 the last and only time that I ran across it.
Yes, I know about those tools. Things like the compiler are 32 bit
(although they produce 16 bit code) and work in a modern system.
Thinks like masm.exe and exe2bin.exe are 16 bit and do not work on
modern, 64 bit systems.
> Personally, while I think the goal of having one build environment obviously makes a lot of sense, I'd just go with gnu tools for this. In particular, yasm (what nasm is called these days, not that either is gnu, strictly speaking) could really make most of this pretty easy for you.
Yes, I used nasm a lot with some UNIX projects. At the moment, I think
nasm or yasm are the best options. I can put the yasm binaries in my
bootloader subdirectory and use a custom Makefile to be used by
build.exe that references yasm.exe as a build tool, so when someone
checkouts the tree and want to build the thing, won't have to install
stuff separately etc. I only need this kind of stuff for the
bootloader.
--
Aram Hăvărneanu
> objcopy --output-target binary <in> <out>
I can use objcopy with binaries generated by the WDK tools, but it's
useless as the code is 32 bit, so I'd still need a 16 bit stub that
does the transition to 32 bit.
--
Aram Hăvărneanu
> In any case, what surprises me here is the OP's choice of WDK as of his build environment for his _own_ OS - this option seems to be quite strange for someone who wants to build a custom OS (unless the target OS somehow relies upon illegitimately obtained Windows sources, of course).....
I think the opposite is true. I mostly use Windows as a desktop OS
this days (I've used UNIX exclusively many years). On Windows, the WDK
is great. You just install it and you don't have to think about the
build tools too much. GCC on Windows is a nightmare. Simply put,
although GCC mostly works on Windows, Windows is not a first tier
platform for GNU. There isn't even a single install. There are many
mingw, cygwin and djgpp versions. Considering djgpp obsolete (16 bit
DOS binaries), mingw and cygwin remain. mingw is never updated and the
quality is lacking. For instance, C++ exceptions don't work across
dlls. This is irrelevant to kernel development, but it gives you an
idea about the quality of the product. Cygwin usually crashes on 64
bit and even when it works, the performance is abysmal.
I've done some work on Solaris and BSDs. Excepting Solaris, which uses
it's own compiler, the BSDs use GCC. They do not use "vanilla" GNU GCC
but they maintain their own fork, which in some cases (OpenBSD) is
very old (OpenBSD used GCC 2.95 (some platforms) and GCC 3.3 (most
platforms)) and very, very patched. Basically they do not blindly
trust GNU GCC. I don't blindly trust GCC either, but I do not have the
resources of studying GCC code to see what versions is "best" and what
do I have to patch, and even if had the resources, I wish to spend
time coding my own product and drinking beer and painting and writing
poetry instead of tinkering with the build tools. I trust Microsoft
release engineering much more then GNU releng.
Apple and FreeBSD want to replace GCC with LLVM+clang, OpenBSD and
NetBSD want to replace GCC with PCC. There is this tendency of
replacing GCC even in UNIX world because of the decreasing quality of
the product and because it is slower and slower and less maintainable.
Why would I want to use GCC on Windows where GCC works even worse then
on UNIX? I think WDK tools represent the *sane" choice. The drawback
is that the WDK tools target only IA-32(e) and IA-64, while GCC
targets... everything.
As for the other thing... my OS can have nothing in common with
Windows code because:
1) My OS is a pure third generation L4 architecture microkernel that
runs most of the stuff in userspace.
2) My OS is Open Source, so anybody can look at the code. I didn't
release any code yet, but I plan to do so at every release. That is,
is "Open Source" but not "Open Development".
--
Aram Hăvărneanu
build.exe to be superiour to GNU Make in every aspect. No wonder *EVERY* OS
(including Linux) has developed it's own build tools and it not using GNU
Make...
I'm happy with build.exe, I want to use it and drink beer, I don't want to
develop an alternative build system. OS devel is hard enough ;-).
--
Aram Hăvărneanu
WDK seems to be (IMHO, of course) quite strange option for someone who wants to build a custom OS simply because I would not expect someone who wants to develop his own OS from the scratch to be willing to sacrifice his flexibility for whatever reason - he hardly needs anything, apart from compiler, linker , assembler and command-line utility that allows him to invoke above mentioned tools with the arguments of his choice so that he can get the maximum possible flexibility and control over the build process. Although I would not expect one to write his own binary-code-generating tools, developing an OS without developing its build system sounds...ugh, " a bit strange"....
However, if the target OS heavily relies upon sources and headers that come from elsewhere, the situation becomes totally different, for understandable reasons....
Anton Bassov
You must have made your post while I was typing mine - indeed, every OS tends to have its own build environment. Therefore, I am quite surprised that you want to take another route( unless you have a good reason for this, of course)....
Anton Bassov
But there was a book called 32bit Operating systems, that had basically all the procedure I followed, and was able to boot from the floppy to a 32bit mode. As Tim pointed out, the module was old _com_ formatted. I don't remember exactly but I tried Tasm ( as was referred) and Masm ( just to see ...). Both were able to do the job, iirc. the loader was exactly 512 byte, stuffed into _0_ sector, and the signature was ( I think) 5A5A or some such.
-pro
On Mar 3, 2010, at 3:39 AM, Aram Hăvărneanu wrote:
> Oh yes, adding to the GNU vs. WDK thing above, I must say that I consider build.exe to be superiour to GNU Make in every aspect. No wonder *EVERY* OS (including Linux) has developed it's own build tools and it not using GNU Make...
>
> I'm happy with build.exe, I want to use it and drink beer, I don't want to develop an alternative build system. OS devel is hard enough ;-).
>
> --
> Aram Hăvărneanu
>
> ---
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
It is a good task, but I think WDK is even legally prohibited from being used in this area, and surely this is not technically supported.
Use GNU tools, or strip the PE file using your own tool to the format understandable by GRUB (if you really want to violate the WDK's license).
--
Maxim S. Shatskih
Windows DDK MVP
[email protected]
http://www.storagecraft.com
news:[email protected]
> It is a good task, but I think WDK is even legally prohibited from being
> used in this area, and surely this is not technically supported.
I have read license.rtf, and have not found anything that would suggest that
using the WDK in this mode is legally prohibited. IANAL, but the only
reference to other operating systems is related to using the *samples* on
other OSes, which seems to be illegal. I certainly would not want to break
the law.
> Use GNU tools, or strip the PE file using your own tool to the format
> understandable by GRUB (if you really want to violate the WDK's license).
Well right now, I am able to boot by PE kernel with Grub's multiboot
support. However, this is very fragile, requires the PE file to be build in
a specific way, with stuff at specific addresses, code size is limited etc.
--
Aram Hăvărneanu
Jan
> -----Original Message-----
> From: [email protected] [mailto:bounce-403783-
> [email protected]] On Behalf Of Aram Havarneanu
> Sent: Wednesday, March 03, 2010 3:33 AM
> To: Windows System Software Devs Interest List
> Subject: Re: [ntdev] Flat Executable Compiler Output
>
> On Wed, Mar 3, 2010 at 5:06 AM, <[email protected]> wrote:
> > In any case, what surprises me here is the OP's choice of WDK as of
> his build environment for his _own_ OS - this option seems to be quite
> strange for someone who wants to build a custom OS (unless the target
> OS somehow relies upon illegitimately obtained Windows sources, of
> course).....
>
> I think the opposite is true. I mostly use Windows as a desktop OS
> this days (I've used UNIX exclusively many years). On Windows, the WDK
> is great. You just install it and you don't have to think about the
> build tools too much. GCC on Windows is a nightmare. Simply put,
> although GCC mostly works on Windows, Windows is not a first tier
> platform for GNU. There isn't even a single install. There are many
> mingw, cygwin and djgpp versions. Considering djgpp obsolete (16 bit
> DOS binaries), mingw and cygwin remain. mingw is never updated and the
> quality is lacking. For instance, C++ exceptions don't work across
> dlls. This is irrelevant to kernel development, but it gives you an
> idea about the quality of the product. Cygwin usually crashes on 64
> bit and even when it works, the performance is abysmal.
>
> I've done some work on Solaris and BSDs. Excepting Solaris, which uses
> it's own compiler, the BSDs use GCC. They do not use "vanilla" GNU GCC
> but they maintain their own fork, which in some cases (OpenBSD) is
> very old (OpenBSD used GCC 2.95 (some platforms) and GCC 3.3 (most
> platforms)) and very, very patched. Basically they do not blindly
> trust GNU GCC. I don't blindly trust GCC either, but I do not have the
> resources of studying GCC code to see what versions is "best" and what
> do I have to patch, and even if had the resources, I wish to spend
> time coding my own product and drinking beer and painting and writing
> poetry instead of tinkering with the build tools. I trust Microsoft
> release engineering much more then GNU releng.
>
> Apple and FreeBSD want to replace GCC with LLVM+clang, OpenBSD and
> NetBSD want to replace GCC with PCC. There is this tendency of
> replacing GCC even in UNIX world because of the decreasing quality of
> the product and because it is slower and slower and less maintainable.
> Why would I want to use GCC on Windows where GCC works even worse then
> on UNIX? I think WDK tools represent the *sane" choice. The drawback
> is that the WDK tools target only IA-32(e) and IA-64, while GCC
> targets... everything.
>
> As for the other thing... my OS can have nothing in common with
> Windows code because:
>
> 1) My OS is a pure third generation L4 architecture microkernel that
> runs most of the stuff in userspace.
> 2) My OS is Open Source, so anybody can look at the code. I didn't
> release any code yet, but I plan to do so at every release. That is,
> is "Open Source" but not "Open Development".
>
> --
> Aram Hăvărneanu
>
> ---
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
> build.exe to be superiour to GNU Make in every aspect.
It is inferior in features, but is is "correct subset wrapper".
Actually, BUILD is a "correct subset wrapper" for NMAKE, which in turn also inferior to GNU make in features.
>No wonder *EVERY* OS
> (including Linux) has developed it's own build tools and it not using GNU
> Make...
Using "correct subset wrappers" for GNU make I think.
--
Maxim S. Shatskih
Windows DDK MVP
[email protected]
http://www.storagecraft.com
IIRC it is renamed Interix (if not even Services for UNIX), but the rest of your post is true.
Historically, the thing used its own PIDs which are _not NT kernel/Win32 PIDs_. This was not so good.
--
Maxim S. Shatskih
Windows DDK MVP
[email protected]
http://www.storagecraft.com
> Apple and FreeBSD want to replace GCC with LLVM+clang.
Yes, it is a very powerful feature - In another discussion on nttalk I said that IL(intermediate langs) has it its own benefit, and I recently encountered this idea pioneered by Latner, it is perhaps to have the Objective-C as front end to gcc ( but it is OT).
-pro
I didn't know that about process id's, although am curious how a kernel process id could be anything other than a kernel process id.
Jan
> -----Original Message-----
> From: [email protected] [mailto:bounce-403817-
> [email protected]] On Behalf Of Maxim S. Shatskih
> Sent: Wednesday, March 03, 2010 7:52 AM
> To: Windows System Software Devs Interest List
> Subject: Re:[ntdev] Flat Executable Compiler Output
>
> >You know, you could run the Windows POSIX subsystem, which I believe
> comes with gcc.
>
> IIRC it is renamed Interix (if not even Services for UNIX), but the
> rest of your post is true.
>
> Historically, the thing used its own PIDs which are _not NT
> kernel/Win32 PIDs_. This was not so good.
>
> --
> Maxim S. Shatskih
> Windows DDK MVP
> [email protected]
> http://www.storagecraft.com
>
>
> ---
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
I just wonder in which jurisdiction one can _legally_ prohibit to use a certain compiler for something other than a certain purpose...
> strip the PE file using your own tool to the format understandable by GRUB (if you really want
> to violate the WDK's license).
Gets more and more exciting....
Are you claiming that one can be _legally_ prohibited from editing binary data if it happens to be a certain compiler's output????
I just wonder what jurisdiction you are living in. I strongly suspect that this "jurisdiction" exists only in
imagination of those who take everything from " Politburo in Redmond" for granted....
Anton Bassov
duplicating the zeros and ones that happen to encode a song or movie.
The same jurisdictions in which you can get free (or nearly free) copies
of Windows and Visual Studio and all the Server products from MSDN,
MSDNAA, or TechNet, but aren't legally allowed to use those bits or the
bits they output for commercial use until you pay more for the exact
same bits.
I thought most jurisdictions respected the value of IP and the legality
of license agreements. Don't assume just because enforcement may be
lax, that it's legal.
In Soviet Russia, software license YOU!
On 3/3/2010 6:09 PM, [email protected] wrote:
>> I think WDK is even legally prohibited from being used in this area,
>>
> I just wonder in which jurisdiction one can _legally_ prohibit to use a certain compiler for something other than a certain purpose...
>
>
>
>
>> strip the PE file using your own tool to the format understandable by GRUB (if you really want
>> to violate the WDK's license).
>>
> Gets more and more exciting....
>
> Are you claiming that one can be _legally_ prohibited from editing binary data if it happens to be a certain compiler's output????
>
> I just wonder what jurisdiction you are living in. I strongly suspect that this "jurisdiction" exists only in
> imagination of those who take everything from " Politburo in Redmond" for granted....
>
>
> Anton Bassov
>
> ---
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>
> of Windows and Visual Studio and all the Server products from MSDN,
> MSDNAA, or TechNet, but aren't legally allowed to use those bits or the
> bits they output for commercial use until you pay more for the exact
> same bits.
No, not the "bits they output".
MSDN license allows installation of any Windows software _for testing purpose_, not for real-world deployment. Nevertheless, MSDN includes 10 usual licenses to client Windows and MS Office.
> In Soviet Russia, software license YOU!
In Soviet Russia, the software was pirated on the level of governemental ministries - mainly the one from RSX-11 ecosystem - there were very popular clones of the machine in the Eastern Bloc, all usually running RSX-11, and there were also nearly-exact - with some hardware customized - clones of desktop LSI-11 running RT-11.
Yes, this was a governement decision - let's reverse-engineer and clone the hardware, and then use pirated software country-wise. The hardware etc was given Russian names (RSX-11 was called The Realtime OS etc), and there were even books on the assembly language of "MiniSystem Electronic Computers" (PDP-11) or "Unified Line of Electronic Computers" (IBM/360), without ever mentioning the original names.
USSR even pirated the whole IBM's vertical solution for selling railroad and airline tickets based on IBM/360.
Well, RT-11 was my first platform to touch when I was a senior schoolboy :-)
--
Maxim S. Shatskih
Windows DDK MVP
[email protected]
http://www.storagecraft.com
> No, not the "bits they output".
>
> MSDN license allows installation of any Windows software _for testing purpose_, not for real-world deployment. Nevertheless, MSDN includes 10 usual licenses to client Windows and MS Office.
>
Academically-licensed Visual Studio Professional generates the exact
same compiler output as commercially-licensed Visual Studio
Professional. Yes, legally, you can't use the "bits they output" for
production use unless you've licensed the compiler for that.
You could say that a test web server deployment licensed under MSDN or
TechNet outputs the same bits over HTTP as a production web server
deployment, but the license agreement says how you can or can't use
those outputted bits.
Semantic issues aside, I was just trying to present analogous examples
wherein the license agreements specify how you can or can't use the bits
generated by the licensed software, and these licenses are (or are in
the US) legally valid and binding. I suppose it would be bad for the
ecosystem in general if the tools could not be licensed in these ways to
different populations who have different needs.
That said, I have no idea what the WDK license agreement says, but it
probably says something about "USER" using "PRODUCT" and a bunch of
other words in all caps.
And in the spirit of Glasnost I'm going to shut up now ^_^
Actually, concerning the latter part (and this is what we are speaking about here) there is very good chance that you are sadly mistaken.
> Don't assume just because enforcement may be lax, that it's legal.
It never occurred to you to think that "enforcement may be lax" simply because there is no legally binding force in such "agreement" in itself??? Some parts of it (for example, concerning distribution) may be, indeed, based upon the existing laws (and well-enforced, BTW), but some may be made up out of nothing and even contradict legislation in some jurisdictions. Just to give you an idea, virtually every EULA in existence disclaims any liability for any damages that may arise from the use of the software. AFAIK, this part would not stand a chance in most (if not in all) jurisdictions....
This is why I was speaking about the "imaginary jurisdiction" that some people seem to be living in....
> In Soviet Russia, software license YOU!
Where is Mr.Kyler???
Anton Bassov