Networking from Native NT Application

Hello.

I have a native NT application that runs at BootExecute time, by which I
mean I place it in the Session Manager’s BootExecute list to be executed
before the page files are initialized or Win32 is fired up.

As if what I’m doing wasn’t messy enough already, the application needs to
use TCP to connect to a network service and pull down some data. Winsock is
right out (Win32 isn’t loaded), and the interface to AFD.SYS doesn’t appear
to be documented anywhere, official or not. So, next on the list of drivers
to abuse is TCPIP.SYS, but from what I’ve been able to discern it’s just not
possible to do network I/O with TCPIP unless you are a kernel-mode TDI
client.

So, there’s the question. Is there any way for a native application running
at BootExecute to access TCP services other than redirecting everything
through a custom TDI client driver? And I guess a question I should be
asking as well is: am I likely to hit problems getting TCPIP.SYS to even
work at this point in the boot process?

Thanks for any insight.

Myk Willis

(Tools | Options | Send | Plain Text … check,

Symbol server set up correctly…check,

!analyze -v output…little too early for that…

Yahoo! account? Yeah, but it’s real…)

“Myk Willis” wrote in message news:xxxxx@ntdev…
>
>
> And I guess a question I should be
> asking as well is: am I likely to hit problems getting TCPIP.SYS to even
> work at this point in the boot process?
>

Ah, THERE’s the rub. You have to get the network stack started. Once you
get the network stack running, you should have little problem if you want to
do something relatively simple with UDP or TCP. There’s even an ancient
trivial example on OSR Online somewhere – Thomas Devine (one of the
regulars on this list) can also doubtless shed light on interfacing to TDI.

So, your main challenge is getting the network started early. I’ve been
through this at least once before, and the results were VERY dependent on
the O/S version. In Win2K I had zero luck. With WinXP I was able to start
the network earlier than normal, but IIRC boot time was not possible.

Peter
OSR

Why not collect the data at shutdown and put them some place under %systemroot%
so they’re available at boot?

Myk Willis wrote:

As if what I’m doing wasn’t messy enough already, the application needs to
use TCP to connect to a network service and pull down some data.


If replying by e-mail, please remove “nospam.” from the address.

James Antognini
Windows DDK MVP

James,

Why not collect the data at shutdown and put them some place under
%systemroot%
so they’re available at boot?

That would be nice. Unfortunately, one of the requirements of the
application is that it not require a reboot for its proper operation. If
I could change that requirement (and perhaps there will be, in the end, no
choice) things would of course be greatly simplified.

Thank you for your suggestion, though.

Myk

“James Antognini” wrote in message
news:xxxxx@ntdev…
>
> Why not collect the data at shutdown and put them some place under
%systemroot%
> so they’re available at boot?
>
> Myk Willis wrote:
>
> > As if what I’m doing wasn’t messy enough already, the application needs
to
> > use TCP to connect to a network service and pull down some data.
>
> –
> If replying by e-mail, please remove “nospam.” from the address.
>
> James Antognini
> Windows DDK MVP
>
>
>
>

Peter,

Thank you very much for your reply.

So, your main challenge is getting the network started early. I’ve been
through this at least once before…

Could you give me some hint of what was involved in this? I did notice that
the call from TCPIP.SYS to TdiProviderReady (on XP) occurs rather a long
time after BootExecute. When I hit a breakpoint on that function,
win32k.sys has already been loaded, and things seem to be getting up to full
swing. (I’m assuming here that TdiProviderReady is the point at which the
stack is ready for action?)

fd5bfd10 fcc6921e TDI!TdiProviderReady
fd5bfd20 fcc69d25 tcpip!DecrInitTimeInterfaces+0x66
[…]

kd> !process 0 0
Image: System […]
Image: smss.exe
Image: csrss.exe
Image: winlogon.exe
Image: services.exe
Image: lsass.exe
Image: svchost.exe

I did hunt around osronline a bit and couldn’t find anything quite relevant.
Perhaps someone else on the list has an idea of where to look for this?

Thomas Devine (one of the
regulars on this list) can also doubtless shed light on interfacing to
TDI.

Thomas was kind enough to reply off-list with some suggestions for
interfacing to TDI. He suggested I might just try to open the \Device\Tcp
from user-mode, since I should be able to specify the EA buffer correctly
from there. I’m going to go off and just try this for the sake of it, but
one immediate question I have is: how can you convince the I/O manager to
send an IRP_MJ_INTERNAL_DEVICE_CONTROL on behalf of a user-mode application
(which I will need to do to send TDI requests)? If it were a normal
IRP_MJ_DEVICE_CONTROL, I could just use ZwDeviceIoControlFile, and of course
_READ and _WRITE are no problem, but I don’t know of any equivilent routine
to effect an _INTERNAL_DEVICE_CONTROL.

Thanks again for your help.

Myk

“Peter Viscarola” wrote in message news:xxxxx@ntdev…
>
>
> “Myk Willis” wrote in message news:xxxxx@ntdev…
> >
> >
> > And I guess a question I should be
> > asking as well is: am I likely to hit problems getting TCPIP.SYS to even
> > work at this point in the boot process?
> >
>
> Ah, THERE’s the rub. You have to get the network stack started. Once you
> get the network stack running, you should have little problem if you want
to
> do something relatively simple with UDP or TCP. There’s even an ancient
> trivial example on OSR Online somewhere – Thomas Devine (one of the
> regulars on this list) can also doubtless shed light on interfacing to
TDI.
>
> So, your main challenge is getting the network started early. I’ve been
> through this at least once before, and the results were VERY dependent on
> the O/S version. In Win2K I had zero luck. With WinXP I was able to
start
> the network earlier than normal, but IIRC boot time was not possible.
>
> Peter
> OSR
>
>
>
>

Humm…

I don’t recall saying that you could open \Device\Tcp from user-mode. In
fact, you can’t because Win32 API doesn’t support CreateFile with extended
arrtibute buffer - necessary (for some obscure reason) to open address and
connection endpoints.

You might be able to do this from a native application using NtCreateFile.
Do understand that I’m totally out of my league when native NT applications
enter the picture, so take my comments with a bottle of salt on this topic.

Thomas

“Myk Willis” wrote in message news:xxxxx@ntdev…
>
> Peter,
>
> > Thomas Devine (one of the
> > regulars on this list) can also doubtless shed light on interfacing to
> TDI.
>
> Thomas was kind enough to reply off-list with some suggestions for
> interfacing to TDI. He suggested I might just try to open the \Device\Tcp
> from user-mode, since I should be able to specify the EA buffer correctly
> from there. I’m going to go off and just try this for the sake of it, but
> one immediate question I have is: how can you convince the I/O manager to
> send an IRP_MJ_INTERNAL_DEVICE_CONTROL on behalf of a user-mode
application
> (which I will need to do to send TDI requests)? If it were a normal
> IRP_MJ_DEVICE_CONTROL, I could just use ZwDeviceIoControlFile, and of
course
> _READ and _WRITE are no problem, but I don’t know of any equivilent
routine
> to effect an _INTERNAL_DEVICE_CONTROL.
>

Myk,

you wrote on Friday, September 5, 2003, 22:53:56:

MW> So, there’s the question. Is there any way for a native application running
MW> at BootExecute to access TCP services other than redirecting everything
MW> through a custom TDI client driver?

Check www.storagecraft.com. They offer a Kernsock package which you can
utilize from your native application to get very much Winsock-like
functionality.

Ralf.

“Thomas F. Divine” wrote in message news:xxxxx@ntdev…
>
> Humm…
>
> I don’t recall saying that you could open \Device\Tcp from user-mode. In
> fact, you can’t because Win32 API doesn’t support CreateFile with extended
> arrtibute buffer - necessary (for some obscure reason) to open address and
> connection endpoints.
>
> You might be able to do this from a native application using NtCreateFile.
> Do understand that I’m totally out of my league when native NT
applications
> enter the picture, so take my comments with a bottle of salt on this
topic.
>

Fortunately, lots of people on the list ARE familiar with native apps in
user mode, so we can probably help a bit…

You CAN use the native API from user mode to open \device\TPC.

Peter
OSR

“Myk Willis” wrote in message news:xxxxx@ntdev…
>
> one immediate question I have is: how can you convince the I/O manager to
> send an IRP_MJ_INTERNAL_DEVICE_CONTROL on behalf of a user-mode
application
> (which I will need to do to send TDI requests)? If it were a normal
> IRP_MJ_DEVICE_CONTROL, I could just use ZwDeviceIoControlFile, and of
course
> _READ and _WRITE are no problem, but I don’t know of any equivilent
routine
> to effect an _INTERNAL_DEVICE_CONTROL.
>

OK, that’s the second part of the question (sorry, I didn’t read this
initially).

Is there any API that will let you send an IRP_MJ_INTERNAL_DEVICE_CONTROL
from user mode? The answer to this is a very resounding “NO”. This is
deliberate.

So, while you can certainly OPEN \device\TCP from user-mode, I guess that
means you can’t USE it (except via AFD, and that’s a whole other “mystery”).

Anyhow, that puts us back in kernel mode.

When we had to start the network stack early on XP, I recall that we changed
its startup time. We were successful in doing this, but I do not believe we
were able to get it to work at boot time. Sigh! I can’t remember the
details. Age must be affecting my memory…

You’re just going to have to fool with it. Fortunately, it shouldn’t be too
difficult an experiment,

Peter

>

You CAN use the native API from user mode to open \device\TPC.

Yes, so I’ve succesfully opened both a transport address and connection
endpoint object from my user mode app (using ZwCreateFile with the EA buffer
set up correctly). So far so good.

Next step: associate my connection endpoint with the transport address.
This is the first thing that requires an actual TDI command
(TDI_ASSOCIATE_ADDRESS).

Peter enlightened me in another post with:

Is there any API that will let you send an IRP_MJ_INTERNAL_DEVICE_CONTROL
from user mode? The answer to this is a very resounding “NO”. This is
deliberate.

Makes sense. But I did later find that the TDI functions are exposed in two
ways: one as minor function codes for IRP_MJ_INTERNAL_DEVICE_CONTROL (the
TDI_XXX values), and also as regular, run of the mill DeviceContorl IOCTLs
of the form IOCTL_TDI_XXX, defined in the NTDDTDI.H header file. TDI
providers apparently use TdiMapUserRequest to map the IOCTL to the INTERNAL
D/C.

So what the heck. I gave it a whirl with the following code
(_SendTdiControl just does a ZwDeviceIoControlFile):

TDI_REQUEST_KERNEL_ASSOCIATE associateAddressParams;

associateAddressParams.AddressHandle = TransportAddressHandle;

status = _SendTdiControl(
ConnectionEndpointHandle,
IOCTL_TDI_ASSOCIATE_ADDRESS,
&associateAddressParams,
sizeof(TDI_REQUEST_KERNEL_ASSOCIATE),
NULL, 0
);

And I get back…(drum roll…): STATUS_NOT_IMPLEMENTED

Same thing with other IOCTLS (although presumably GET_INFORMATION works as
I’ve seen examples of people using that on the “\Device\Tcpip” device opened
from Win32). It looks like the IOCTL interface is blocked, at least from
user mode, for whatever reason.

So, I think you’re right. I’m back to kernel mode, and can now focus my
energy on trying to figure out how I might load the network stack early.

Thanks Peter and Thomas for your help.

Myk

“Peter Viscarola” wrote in message news:xxxxx@ntdev…
>
>
> “Thomas F. Divine” wrote in message
news:xxxxx@ntdev…
> >
> > Humm…
> >
> > I don’t recall saying that you could open \Device\Tcp from user-mode. In
> > fact, you can’t because Win32 API doesn’t support CreateFile with
extended
> > arrtibute buffer - necessary (for some obscure reason) to open address
and
> > connection endpoints.
> >
> > You might be able to do this from a native application using
NtCreateFile.
> > Do understand that I’m totally out of my league when native NT
> applications
> > enter the picture, so take my comments with a bottle of salt on this
> topic.
> >
>
> Fortunately, lots of people on the list ARE familiar with native apps in
> user mode, so we can probably help a bit…
>
> You CAN use the native API from user mode to open \device\TPC.
>
> Peter
> OSR
>
>
>
>

Thank, Ralf. I noticed OSR has such a package (KSOCKS) as well. I may
indeed go that way if I can get the network stack started early enough.

Myk

“Ralf Buschmann” wrote in message news:xxxxx@ntdev…
>
> Myk,
>
> you wrote on Friday, September 5, 2003, 22:53:56:
>
> MW> So, there’s the question. Is there any way for a native application
running
> MW> at BootExecute to access TCP services other than redirecting
everything
> MW> through a custom TDI client driver?
>
> Check www.storagecraft.com. They offer a Kernsock package which you can
> utilize from your native application to get very much Winsock-like
> functionality.
>
> Ralf.
> –
>
>
>

Trying to do TCP work at shutdown is another goodie.

If the power IRP have already passed through NDIS’s device object for the NIC -
then the packets are blocked in the NDIS’s queue, and the other end will never
see your “goodbye” FIN packet.

Trying to do something more complex then closesocket() in shutdown path is
possibly even funnier.

Maybe TdiRegisterPnPHandlers is a solution.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Myk Willis”
Newsgroups: ntdev
To: “Windows System Software Developers Interest List”
Sent: Saturday, September 06, 2003 5:03 AM
Subject: [ntdev] Re: Networking from Native NT Application

> James,
>
> > Why not collect the data at shutdown and put them some place under
> %systemroot%
> > so they’re available at boot?
>
> That would be nice. Unfortunately, one of the requirements of the
> application is that it not require a reboot for its proper operation. If
> I could change that requirement (and perhaps there will be, in the end, no
> choice) things would of course be greatly simplified.
>
> Thank you for your suggestion, though.
>
> Myk
>
> “James Antognini” wrote in message
> news:xxxxx@ntdev…
> >
> > Why not collect the data at shutdown and put them some place under
> %systemroot%
> > so they’re available at boot?
> >
> > Myk Willis wrote:
> >
> > > As if what I’m doing wasn’t messy enough already, the application needs
> to
> > > use TCP to connect to a network service and pull down some data.
> >
> > –
> > If replying by e-mail, please remove “nospam.” from the address.
> >
> > James Antognini
> > Windows DDK MVP
> >
> >
> >
> >
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com

> Check www.storagecraft.com. They offer a Kernsock package which you can

utilize from your native application to get very much Winsock-like
functionality.

Yes, and we even have a kerne-mode/user-mode combo which exposes KernSock to
user mode (used for testing initially).

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

I thought the whole problem was the doing of network operations at boot time.
So we are talking about rebooting, at least after your kernel app is installed.
But I must be missing a point, so please enlighten me.

Myk Willis wrote:

Unfortunately, one of the requirements of the
application is that it not require a reboot for its proper operation.


If replying by e-mail, please remove “nospam.” from the address.

James Antognini
Windows DDK MVP

James,

The product does a kind of recovery. It is installed on a machine, it does
‘normal’ processing using Win32 applications with well-behaved APIs and so
foth, and time passes. At some point in the future it is “called into
action,” potentially by a special boot sequence that ends up with the native
application running at BootExecute. At this point, it takes some
information it has squirreled away in a special cache and processes it by
writing to various files. It’s quite a bit more complicated than simply
copying files, though, so standard tricks for backup applications don’t
really apply.

So this is fine, and it works today. But now there is a desire to be able
to pull that cached information not from the local filesystem, but instead
from the network. Which takes us to last week, where I’m trying to figure
out if I can actually use any network services from BootExecute or not.
I’ve found by experiment (and with the help of this list) that I can’t write
a user-mode network client, so now I have to experiement with whether a
kernel mode TDI client can function this early.

I just want to exhaust all design possibilities for the client’s
architecture before suggesting that they may have to change it.

Myk

“James Antognini” wrote in message
news:xxxxx@ntdev…
>
> I thought the whole problem was the doing of network operations at boot
time.
> So we are talking about rebooting, at least after your kernel app is
installed.
> But I must be missing a point, so please enlighten me.
>
> Myk Willis wrote:
>
> > Unfortunately, one of the requirements of the
> > application is that it not require a reboot for its proper operation.
>
> –
> If replying by e-mail, please remove “nospam.” from the address.
>
> James Antognini
> Windows DDK MVP
>
>
>
>

“Special boot sequence”? Have you considered running DOS first, to gather the
information and put into a local file and then booting to Windows? That’d
probably be messy, but I suppose it could be made to work.

That idea aside, the basic problem is that there is no guarantee of just when
networking becomes available. You might have it when you need it, with a certain
OS level and service level, and later you might not have it. I imagine you could
make the starting of your driver contingent on the starting of TCPIP (or
whatever), but you’d still need to wait around until TCPIP or whatever is fully
up. And even in that case, it might be too late to do what you want to do.

What is it, by the way, that you’re actually trying to accomplish? What’s the
use of the cached information?

Myk Willis wrote:

At some point in the future it is “called into
action,” potentially by a special boot sequence that ends up with the native
application running at BootExecute. At this point, it takes some
information it has squirreled away in a special cache and processes it by
writing to various files.

[snip]

But now there is a desire to be able
to pull that cached information not from the local filesystem, but instead
from the network.


If replying by e-mail, please remove “nospam.” from the address.

James Antognini
Windows DDK MVP

Myk,

you wrote on Saturday, September 6, 2003, 21:04:40:

> Check www.storagecraft.com. They offer a Kernsock package which you can
> utilize from your native application to get very much Winsock-like
> functionality.

MW> Thank, Ralf. I noticed OSR has such a package (KSOCKS) as well. I may
MW> indeed go that way if I can get the network stack started early enough.

As far as I can tell, the network stack is started at “BootExecute
time”. We have a Kernsock license and I have successfully used it to
connect to an external FTP sever in a native application. Kernsock
relies only on TDI. I didn’t mess with the ‘Start’ registry parameter of
any network services, I just installed the Kernsock driver and then from
the native app you send a custom IOCTL to the Kernsock driver to get a
socket handle on an IP address and then you can use
NtReadFile()/NtWriteFile() to talk over the socket.

Ralf.