Native API differences ...

Hi,
? ? ? Staring from older versions of ?Windows ( ?Win2000 ? ) ?to ?the latest versions of windows (Windows 8.1 ) there should have been additions?
and deletions of Kernel APIs( I think it can also be called as native APIs ) in each new version windows…

Are these differences documented anywhere ? ?or any other way to know this indirectly ? ( I thought one way it could be done is , to find the differences in wdm.h or?ntddk.h in each version of the DDK / WDK??) …Any other way ?

Thanks & Regards,
Anitha

Well the native API’s that are documented by Microsoft are pretty stable,
but that is a small percentage of the total native interface. For the rest
there is no document identifying the differences that I know of. Worse yet
there are still a ton of samples out there that use API’s that are
undocumented and have change data structures used for input or output, and
even worse the meaning of the class codes for the query/set operations which
people blindly follow over the cliff.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Anitha Govindarajan
Sent: Wednesday, September 11, 2013 1:10 PM
To: Windows File Systems Devs Interest List
Subject: [ntfsd] Native API differences …

Hi,
Staring from older versions of Windows ( Win2000 ) to the latest
versions of windows (Windows 8.1 ) there should have been additions
and deletions of Kernel APIs( I think it can also be called as native APIs )
in each new version windows…

Are these differences documented anywhere ? or any other way to know this
indirectly ? ( I thought one way it could be done is , to find the
differences in wdm.h or ntddk.h in each version of the DDK / WDK ) …Any
other way ?

Thanks & Regards,
Anitha


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

you may find some information regarding which function appeared in
which kernel and which disappeared in this link (i dont know if it is
thorough or not i found it in one of my regular sweeps and bookmarked
it that is all)

http://www.geoffchappell.com/studies/windows/km/ntoskrnl/api/index.htm

On 9/11/13, Don Burn wrote:
> Well the native API’s that are documented by Microsoft are pretty stable,
> but that is a small percentage of the total native interface. For the rest
> there is no document identifying the differences that I know of. Worse yet
> there are still a ton of samples out there that use API’s that are
> undocumented and have change data structures used for input or output, and
> even worse the meaning of the class codes for the query/set operations
> which
> people blindly follow over the cliff.
>
>
> Don Burn
> Windows Filesystem and Driver Consulting
> Website: http://www.windrvr.com
> Blog: http://msmvps.com/blogs/WinDrvr
>
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Anitha
> Govindarajan
> Sent: Wednesday, September 11, 2013 1:10 PM
> To: Windows File Systems Devs Interest List
> Subject: [ntfsd] Native API differences …
>
> Hi,
> Staring from older versions of Windows ( Win2000 ) to the
> latest
> versions of windows (Windows 8.1 ) there should have been additions
> and deletions of Kernel APIs( I think it can also be called as native APIs
> )
> in each new version windows…
>
> Are these differences documented anywhere ? or any other way to know this
> indirectly ? ( I thought one way it could be done is , to find the
> differences in wdm.h or ntddk.h in each version of the DDK / WDK ) …Any
> other way ?
>
> Thanks & Regards,
> Anitha
>
> —
> NTFSD is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTFSD is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

While a nice site, this does not distinguish the changes. For instance many
people believe the ZwXXXSystemInformation calls that are documented on the
web have stayed constant, this is far from the case, including the
capability to load a driver has had radical changes to the semantics.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of raj_r
Sent: Wednesday, September 11, 2013 2:54 PM
To: Windows File Systems Devs Interest List
Subject: Re: [ntfsd] Native API differences …

you may find some information regarding which function appeared in which
kernel and which disappeared in this link (i dont know if it is thorough or
not i found it in one of my regular sweeps and bookmarked it that is all)

http://www.geoffchappell.com/studies/windows/km/ntoskrnl/api/index.htm

On 9/11/13, Don Burn wrote:
> Well the native API’s that are documented by Microsoft are pretty
> stable, but that is a small percentage of the total native interface.
> For the rest there is no document identifying the differences that I
> know of. Worse yet there are still a ton of samples out there that
> use API’s that are undocumented and have change data structures used
> for input or output, and even worse the meaning of the class codes for
> the query/set operations which people blindly follow over the cliff.
>
>
> Don Burn
> Windows Filesystem and Driver Consulting
> Website: http://www.windrvr.com
> Blog: http://msmvps.com/blogs/WinDrvr
>
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Anitha
> Govindarajan
> Sent: Wednesday, September 11, 2013 1:10 PM
> To: Windows File Systems Devs Interest List
> Subject: [ntfsd] Native API differences …
>
> Hi,
> Staring from older versions of Windows ( Win2000 ) to the
> latest
> versions of windows (Windows 8.1 ) there should have been additions
> and deletions of Kernel APIs( I think it can also be called as native
> APIs
> )
> in each new version windows…
>
> Are these differences documented anywhere ? or any other way to know
> this indirectly ? ( I thought one way it could be done is , to find
> the differences in wdm.h or ntddk.h in each version of the DDK / WDK
> ) …Any other way ?
>
> Thanks & Regards,
> Anitha
>
> —
> NTFSD is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTFSD is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

The “native API” for the most part is “undocumented”. This is vendorspeak
for “we reserve the right, at any time, to make changes, even breaking
changes, and if your code stops working, well, too bad”

Using the undocumented native API is extremely risky. It can change
incompatibly with a hotfix. It comes with no guarantees. So if you find
documentation, it is at best a snapshot of someone’s attempt to
reverse-engineer the behavior of a call. There is no guarantee the person
has done a thorough, or even correct, job, and the actual behavior may
critically depend on some OS state the person documenting it is unaware
of.

There are no guarantees for undocumented interfaces. All existing
documentation for the otherwise-undocumented interfaces is suspect. Use
at your own risk. Void where prohibited by law. Does not include dealer
prep fees. Cash value 1/20th of one cent. Your mileage may vary.
joe

Hi,
      Staring from older versions of  Windows (  Win2000   )  to  the
latest versions of windows (Windows 8.1 ) there should have been
additions 
and deletions of Kernel APIs( I think it can also be called as native APIs
) in each new version windows…

Are these differences documented anywhere ?  or any other way to know this
indirectly ? ( I thought one way it could be done is , to find the
differences in wdm.h or ntddk.h in each version of the DDK / WDK  ) …Any
other way ?

Thanks & Regards,
Anitha

NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

On 09/11/2013 07:47 PM, xxxxx@flounder.com wrote:

The “native API” for the most part is “undocumented”. This is vendorspeak
for “we reserve the right, at any time, to make changes, even breaking
changes, and if your code stops working, well, too bad”

Using the undocumented native API is extremely risky. It can change
incompatibly with a hotfix. It comes with no guarantees. So if you find
documentation, it is at best a snapshot of someone’s attempt to
reverse-engineer the behavior of a call. There is no guarantee the person
has done a thorough, or even correct, job, and the actual behavior may
critically depend on some OS state the person documenting it is unaware
of.
While all of these (and the previous arguments) are perfectly valid, one
should also keep in mind that Microsoft uses some of these APIs in tools
that are distributed detached from the OS. So a hotfix would kill these
tools and MS isn’t taking backward-compatibility lightly.

So while one should be cautious with the undocumented parts and have a
plan B in store, if you verify that the function(s) you want to use are
used by MS-sanctioned tools that are not part of the OS-distribution,
chances are that you are safe w.r.t. potential changes. It’s not 100%,
but it’s good enough in many cases.

And in defense of the use of native APIs, there are cases where MS
*claims* in their docs - NtDeviceIoControlFile for me being the *prime*
example - that a Win32 function (here DeviceIoControl) *supersedes* the
“deprecated” native API function. Which in this case couldn’t be further
from the truth. NtDeviceIoControlFile gives you access to the
IoStatusBlock, which DeviceIoControl simply neglects. This can be very
relevant for many an I/O operation.

Aside from that the “native API” includes also those parts we commonly
deal with as driver writers when we don’t make use of some convenience
framework (“legacy drivers”). Ke*, Nt*, Ex*, Mm*, Cm*, Cc*, Ob*, … you
know them.

As long as you don’t rely on SSDT indexes or something silly, I hold
that the native API can be used safely when the user is aware of the
risks and takes measures to fail gracefully should they misbehave.

That said, this link may be what you are looking for … approximately:

http:</http:>

Your other option would be to collect all ntoskrnl.exe, hal.dll,
ntdll.dll (and variations) from the different OS versions, SPs, bitness,
and use something like the pefile Python module to parse the exports.
Assuming you’re interested in the “public” parts. With the private
symbols you’d have to parse debug symbols as well.

Good luck.

// Oliver

> from the truth. NtDeviceIoControlFile gives you access to the

IoStatusBlock, which DeviceIoControl simply neglects.

First 2 fields of OVERLAPPED are the IOSB.

What NtDeviceIoControlFile really provides is the ability to use APC-based completion, like in ReadFileEx.

Too bad there is no kernel32!DeviceIoControlEx

Also, I’m really missing the GetDiskFreeSpaceExByHandle function. That’s why I use NtQueryVolumeInformationFile instead :slight_smile:


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

That gives you the table of the calls, that is trivially easy to acquire with a little bit of dumping of ntdll.sys. Unfortunately, knowing what the parameters are is another question entirely. And as I have pointed out the parameters can change fairly often in Windows.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Oliver Schneider
Sent: Wednesday, September 11, 2013 5:24 PM
To: Windows File Systems Devs Interest List
Subject: Re: [ntfsd] Native API differences …

On 09/11/2013 07:47 PM, xxxxx@flounder.com wrote:

The “native API” for the most part is “undocumented”. This is
vendorspeak for “we reserve the right, at any time, to make changes,
even breaking changes, and if your code stops working, well, too bad”

Using the undocumented native API is extremely risky. It can change
incompatibly with a hotfix. It comes with no guarantees. So if you
find documentation, it is at best a snapshot of someone’s attempt to
reverse-engineer the behavior of a call. There is no guarantee the
person has done a thorough, or even correct, job, and the actual
behavior may critically depend on some OS state the person documenting
it is unaware of.
While all of these (and the previous arguments) are perfectly valid, one should also keep in mind that Microsoft uses some of these APIs in tools that are distributed detached from the OS. So a hotfix would kill these tools and MS isn’t taking backward-compatibility lightly.

So while one should be cautious with the undocumented parts and have a plan B in store, if you verify that the function(s) you want to use are used by MS-sanctioned tools that are not part of the OS-distribution, chances are that you are safe w.r.t. potential changes. It’s not 100%, but it’s good enough in many cases.

And in defense of the use of native APIs, there are cases where MS
*claims* in their docs - NtDeviceIoControlFile for me being the *prime* example - that a Win32 function (here DeviceIoControl) *supersedes* the “deprecated” native API function. Which in this case couldn’t be further from the truth. NtDeviceIoControlFile gives you access to the IoStatusBlock, which DeviceIoControl simply neglects. This can be very relevant for many an I/O operation.

Aside from that the “native API” includes also those parts we commonly deal with as driver writers when we don’t make use of some convenience framework (“legacy drivers”). Ke*, Nt*, Ex*, Mm*, Cm*, Cc*, Ob*, … you know them.

As long as you don’t rely on SSDT indexes or something silly, I hold that the native API can be used safely when the user is aware of the risks and takes measures to fail gracefully should they misbehave.

That said, this link may be what you are looking for … approximately:

http:</http:>

Your other option would be to collect all ntoskrnl.exe, hal.dll, ntdll.dll (and variations) from the different OS versions, SPs, bitness, and use something like the pefile Python module to parse the exports.
Assuming you’re interested in the “public” parts. With the private symbols you’d have to parse debug symbols as well.

Good luck.

// Oliver


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

On 09/11/2013 09:37 PM, Don Burn wrote:

That gives you the table of the calls, that is trivially easy to
acquire with a little bit of dumping of ntdll.sys.
Assuming you can get all the files, which is not so trivial after all,
especially for older ones.

Unfortunately, knowing what the parameters are is another question
entirely.
I may have misread the question, since I took the additions/deletions as
referring to the availability of the API (in the sense of
function/system call), not the parameters. Sorry.

// Oliver

On 09/11/2013 09:34 PM, Maxim S. Shatskih wrote:

First 2 fields of OVERLAPPED are the IOSB.
Wow, thanks! Didn’t realize that.

What NtDeviceIoControlFile really provides is the ability to use APC-based completion, like in ReadFileEx.
Yep, true point, but most casual developers don’t even know about APC.

Also, I’m really missing the GetDiskFreeSpaceExByHandle function. That’s why I use NtQueryVolumeInformationFile instead :slight_smile:
Yep, there are lots of cases were, despite the risks, the use of the
native API is merited.

// Oliver

But the distinguishing thing about some of the api’s such as NtQueryVolumeInformationFile is they are documented by Microsoft since the Zw version of the call is reasonably documented. Of course some of the calls that are documented are only partially documented in that there are query class values that Microsoft does not explain. The real danger is wandering into the areas where things are undocumented, and therefore likely to change, but that seemed to be OP original question.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Oliver Schneider
Sent: Wednesday, September 11, 2013 6:04 PM
To: Windows File Systems Devs Interest List
Subject: Re: [ntfsd] Native API differences …

On 09/11/2013 09:34 PM, Maxim S. Shatskih wrote:

First 2 fields of OVERLAPPED are the IOSB.
Wow, thanks! Didn’t realize that.

What NtDeviceIoControlFile really provides is the ability to use APC-based completion, like in ReadFileEx.
Yep, true point, but most casual developers don’t even know about APC.

Also, I’m really missing the GetDiskFreeSpaceExByHandle function.
That’s why I use NtQueryVolumeInformationFile instead :slight_smile:
Yep, there are lots of cases were, despite the risks, the use of the native API is merited.

// Oliver


NTFSD is sponsored by OSR

OSR is hiring!! Info at http://www.osr.com/careers

For our schedule of debugging and file system seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

>> First 2 fields of OVERLAPPED are the IOSB.

Wow, thanks! Didn’t realize that.

Internal

The status code for the I/O request

An NTSTATUS code, actually Iosb.Status

InternalHigh

The number of bytes transferred for the I/O request.

This is Iosb.Information

kernel32 just casts OVERLAPPED* to IO_STATUS_BLOCK* and passes it as UserIosb.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

> On 09/11/2013 07:47 PM, xxxxx@flounder.com wrote:

> The “native API” for the most part is “undocumented”. This is
> vendorspeak
> for “we reserve the right, at any time, to make changes, even breaking
> changes, and if your code stops working, well, too bad”
>
> Using the undocumented native API is extremely risky. It can change
> incompatibly with a hotfix. It comes with no guarantees. So if you
> find
> documentation, it is at best a snapshot of someone’s attempt to
> reverse-engineer the behavior of a call. There is no guarantee the
> person
> has done a thorough, or even correct, job, and the actual behavior may
> critically depend on some OS state the person documenting it is unaware
> of.
While all of these (and the previous arguments) are perfectly valid, one
should also keep in mind that Microsoft uses some of these APIs in tools
that are distributed detached from the OS. So a hotfix would kill these
tools and MS isn’t taking backward-compatibility lightly.

See the thread about GetVersionEx before making such a statement

So while one should be cautious with the undocumented parts and have a
plan B in store, if you verify that the function(s) you want to use are
used by MS-sanctioned tools that are not part of the OS-distribution,
chances are that you are safe w.r.t. potential changes. It’s not 100%,
but it’s good enough in many cases.

“Good enough” != “Guaranteed”

And in defense of the use of native APIs, there are cases where MS
*claims* in their docs - NtDeviceIoControlFile for me being the *prime*
example - that a Win32 function (here DeviceIoControl) *supersedes* the
“deprecated” native API function. Which in this case couldn’t be further
from the truth. NtDeviceIoControlFile gives you access to the
IoStatusBlock, which DeviceIoControl simply neglects. This can be very
relevant for many an I/O operation.

Sort of curious…why? There are only two values in the IoStatus, one of
which is the status, which is returned in translated form as the
GetLastError value, and one of which is the Information field, which is
returned as the BytesTransferred value. So what is in that block, other
than the untranslated error code, that is of interest?

Aside from that the “native API” includes also those parts we commonly
deal with as driver writers when we don’t make use of some convenience
framework (“legacy drivers”). Ke*, Nt*, Ex*, Mm*, Cm*, Cc*, Ob*, … you
know them.

As long as you don’t rely on SSDT indexes or something silly, I hold
that the native API can be used safely when the user is aware of the
risks and takes measures to fail gracefully should they misbehave.

That is asking a lot. Define “misbehave”, for example. See the other
posts about how some values have changed with service packs. How do I
really deal with values that are completely different from the expected
values?

That said, this link may be what you are looking for … approximately:

http:</http:>

Your other option would be to collect all ntoskrnl.exe, hal.dll,
ntdll.dll (and variations) from the different OS versions, SPs, bitness,
and use something like the pefile Python module to parse the exports.
Assuming you’re interested in the “public” parts. With the private
symbols you’d have to parse debug symbols as well.

I have written some tools that use the native API. I have no expectation
that they will function correctly version-to-version, although they worked
through Vista even though they were developed on Win2K. I consider that
to be mere good fortune.

BTW, it is worth pointing out that the trick given by Nebbit for putting
the native symbols in a namespace, e.g.

namespace Whatever {
#include <ntddk.h>
}

stopped working some releases back. If used, you get fatal compilation
errors while it is reading the header file; I had to add #ifndef
NEBBIT/#endif in the standard header file and define NEBBIT before doing
the #include.

joe
>
> Good luck.
>
> // Oliver
>
> —
> NTFSD is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
></ntddk.h>

Responding to two mails in one. Hope you don’t mind.

On 09/11/2013 10:20 PM, Don Burn wrote:

The real danger is wandering into the areas where things are
undocumented, and therefore likely to change, but that seemed to be
OP original question.
This is what I don’t understand. No statement was made what the
information would be used for. And again, while a cautionary note is
certainly in order, the OP needs to weigh the risks against the
benefits, no?

On 09/12/2013 11:57 AM, xxxxx@flounder.com wrote:>> On 09/11/2013

See the thread about GetVersionEx before making such a statement
Would you be so kind to point it out to me (perhaps a link to the list
archive). For the life of me I couldn’t find it, despite having an
archive reaching back nearly ten years.

“Good enough” != “Guaranteed”
Agreed. And still it’s on the OP (or his/her supervisor) to make the
ultimate decision.

Sort of curious…why? There are only two values in the IoStatus,
one of which is the status, which is returned in translated form as
the GetLastError value, and one of which is the Information field,
which is returned as the BytesTransferred value. So what is in that
block, other than the untranslated error code, that is of interest?
Two things, but Maxim already made it clear that the information can be
obtained. So, as always when confronted with new facts, I reserve the
right to change my mind. It’s obvious that all information is there.

Oh yeah, and what I meant (under my assumption of the info being
unavailable) is that NTSTATUS can be more useful than Win32 error codes
in many cases, because all that matters is the form, the information
carried need not concern some “converter” function in ntdll.dll.

That is asking a lot. Define “misbehave”, for example. See the
other posts about how some values have changed with service packs.
How do I really deal with values that are completely different from
the expected values?
For starters you could set a function pointer depending on the SP level
and OS version, so that inside one piece of code all the known versions
of the OS can be covered.

All I’m saying is that despite the cautionary tales we have to tell and
the advice we have to give about the risks, we cannot completely
discount the use of the native API for specific purposes [*].

The risk is on the OP. The one thing that concerns me most is that this
is being asked on a list about kernel mode development. But some people
like blue :wink:

// Oliver

[*] the OP wasn’t specific enough to give more than generic advice, though.