Strange problem with TDI

Hi,

I’m having a problem with my driver. For other reasons that don’t matter I have to write a tcp server that accepts connections from a regular tcp client application. It has to work on win2k/xp/2k3 so the obvious choice was TDI.

I suspect there’s a problem with my design in the way I associate transport addresses and connection contexts. What I basicaly need to do is to convert the semantics of TDI to something very similar to user-mode socket semantics. Just to take the record straight, from this point when I say “socket” I don’t mean user-mode socket, I mean my kernel-mode implementation of a network socket which has very similar semantics with standard berkeley sockets. I know it’s not quite right to call them “sockets” but I don’t know how else to call them.

Back to the theory: My driver is supposed to accept tcp connections. This means I must have a listening “socket”. This pseudo-“socket” is built from a transport address and a connection context that are linked and the connection context is set to accept incoming connections. To be able to serve those incoming connections I also have a backlog of other “sockets” that are made from other pairs of connection context / transport address but those also have a pre-allocated data buffer used to dump the data I will later get from the remote applications. Each of those backlog “sockets” also have a preallocated Accept Irp that would be later used. When I get the ClientEventConnect notification from TDI I would pick one “socket” from the backlog and tell TDI to use it as a local endpoint for the incoming connection. I do by passing the Accept IRP and the custom connection context of the backlog “socket”. After a connection is received and it’s accepted I create a new “socket” to replenish my backlog that is bound to the same ip/port with the one that has just been used.

This design works, I tested many edge cases and it was ok except one. Let’s say I have a listening “socket”, two “sockets” in my backlog reserve and one active “socket” that is actualy connected to a remote application. This active/connected “socket” had been a backlog “socket” before the connection was enstablished and was spawned from the same listening socket that is still waiting for incoming connections. At some point I wanna stop listening for new incoming connections but still keep my connected “socket” with the remote party alive. Obviously I close the listening and the backlog “sockets” by closing their connection contexts and transport addresses. At this point the connection with the remote party is lost and the remote application receives the disconnect reason “dropped by remote party”. In theory this should not have happened because the connection contexts of the closed “sockets” are independent of the one of the active one that should have remained opened, and the transport address objects are also different (even though all the transport addresses are referencing the same ip/port).

My theory: I suspect there might be a problem with the transport addresses. I know that one transport address can actualy be associated with multiple connection contexts but in my case there’s one transport address for each connection context and some of the transport addresses overllap (same ip/port). This means that the listening “socket”, all the backlog “sockets” of that listening “socket” and the connected “sockets” spawned from the same listening “socket” would have different transport address objects that reference the same ip/port. Could it be possible that when I close one transport address object the tcp/ipv4 provider would actualy also close all other transport address objects using the same ip/port. The object itself is still valid (for the remaining “socket” that was connected to the remote party) but the connection is severed, the connection does not appear anymore in netstat, it’s droped for sure. A possible fix to this possible design issue would be to have only one transport address object bound to all connection contexts that would use it. This means that all the “sockets” bound to the same ip/port would share the same transport address object, which translates in that fact that all connections contexts of those “sockets” would be associated with the same transport address object referincing the same ip/port. This also means that a new low level layer that manages unique transport address objects should be put implemented in my design but this should not be a problem.

I’m not sure my theory is right and I don’t realy wanna start making extensive modifications to the code only to find out that I might have been wrong. That’s why I’m asking for help. Is my evaluation correct? Does the fix seem valid for my problem? Maybe I got it completely wrong, if I did so please correct me.

Thanks in advance!

Come on guys, does anyone have an idea?

>My theory: I suspect there might be a problem with the transport addresses.

Yes, something like this.

The design you described worked for me for sure in 2001, when I wrote kernel sockets over TDI.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

First of all thanks for your reply.

What was exactly the design that worked? The one with many transport address objects or the one with shared transport address objects between connection contexts?

The concept of replying to the email address ‘evildeathmaster666’ was
too much of a hurdle to overcome. Try a slightly less death metal 12
year old name.

Mark Roddy

On Mon, Mar 1, 2010 at 8:43 AM, wrote:
> Come on guys, does anyone have an idea?
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>

> What was exactly the design that worked? The one with many transport address objects or the one

with shared transport address objects between connection contexts?

Forgot this.

But yes, listen backlog, with pre-created sockets.

listen() maps to ClientEventConnect SET_EVENT_HANDLER.

No TDI_LISTEN at all.

In ClientEventConnect, find the one from backlog, initialize the TDI_ACCEPT IRP, register a completion routine in it, and return the IRP as OUT parameter from ClientEventConnect.

Then the completion routine is fired for TDI_ACCEPT, this is the event that the connection is fully set up.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

Ok thanks. In that case I suppose I’ll implement a cache of transport address objects and it should all work out.

It seems I made a terrible error in my judgement. I can not use only one transport address object and share it between all my sockets because the event handlers are set per transport address not per connection context. It would be impossible to distinquish between the sockets in my ClientEventReceive(). After all it seems that the only viable solution would be to have one transport address object per socket.

But the problem still remains. When the I close the transport address of the listening socket all the connections that were spawned from that socket/transport address are immediately closed. I did some experimented and here’s what I found out so far. If I close any of the spawned sockets everything works as intended, the other spawned sockets remain connected, the listening socket is still listening. But when I close the listening socket it automaticaly closes the spawned ones as well. It seems that the listening socket transport address object has some sort of influence over the ones that were spawned from it. I found no reference of such behaviour on the internet or WDK. I also tried another experiment and when I close the listening socket I don’t destroy the transport address object, I closed it manualy only after all the spawned connections have been shutdown by their remote clients. Guess what? I worked! The spawned connections would not be automaticaly disconnected anymore and the listening socket would stop listening, just as intended.

But it doesn’t make any sense for me, why would the transport address object of the original listening socket behave like a “parent” object for the transport address objects of the spawned sockets?!? Is this what am I supposed to do: maintain the object opened until all its “children” are no longer needed?

Of course it is possible to distinguish between clients - that is what the
ConnectionContext is for.

And no, you do not create one transport address per ‘socket’. You create
one transport address for the address you are listening on and create
multiple transport connections associated with that address.

Stop thinking about this as sockets. TDI is not sockets. Sockets can be
built from TDI of course because AFD and the WINSOCK32.DLL provider do that.

By what is the point in kernel mode where TDI is the model of trying to
hammer it back into that model? It sounds to me like you are just taxing
yourself and the transport with about twice as many objects as you really
need.

Look, sockets are an abstraction. TDI is an abstraction. In a driver
written to work over TDI, I would suggest strongly that you use the TDI
abstraction as the basis for your connection management.

Good Luck,
Dave Cattley

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@yahoo.com
Sent: Tuesday, March 02, 2010 5:16 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Strange problem with TDI

It seems I made a terrible error in my judgement. I can not use only one
transport address object and share it between all my sockets because the
event handlers are set per transport address not per connection context. It
would be impossible to distinquish between the sockets in my
ClientEventReceive(). After all it seems that the only viable solution would
be to have one transport address object per socket.

But the problem still remains. When the I close the transport address of the
listening socket all the connections that were spawned from that
socket/transport address are immediately closed. I did some experimented and
here’s what I found out so far. If I close any of the spawned sockets
everything works as intended, the other spawned sockets remain connected,
the listening socket is still listening. But when I close the listening
socket it automaticaly closes the spawned ones as well. It seems that the
listening socket transport address object has some sort of influence over
the ones that were spawned from it. I found no reference of such behaviour
on the internet or WDK. I also tried another experiment and when I close the
listening socket I don’t destroy the transport address object, I closed it
manualy only after all the spawned connections have been shutdown by their
remote clients. Guess what? I worked! The spawned connections would not be
automaticaly disconnected anymore and the listening socket would stop
listening, just as intended.

But it doesn’t make any sense for me, why would the transport address object
of the original listening socket behave like a “parent” object for the
transport address objects of the spawned sockets?!? Is this what am I
supposed to do: maintain the object opened until all its “children” are no
longer needed?


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

David first of all thanks for your reply.

As i said in my original post by “socket” I mean my implementation of something similar to berkeley sockets in kernel mode. I know they are not realy sockets I just call them that way.

And then you are right and you are right. I am trying to emulate berkeley sockets behaviour in my implementation because for windows vista/7 I will have to use WSK instead of TDI. This means that I must have an abstraction layer that handles network communications over TDI or WSk depending on the platform (2k/xp/2k3 or vista/2008/7). This abstraction layer is actualy exposing an interface much like the user-mode sockets because WSK is very close to that anyway. This means the hard part comes in emulating part of the functionality in AFD. And yup I have one transport address object per each connection context object in the current implementation, just as you said.

After taking a second look at the TDI notification functions: ClientEventConnect(), ClientEventReceive() and ClientEventDisconnect() I think you’re right. For ClientEventReceive() and ClientEventDisconnect() I get both TdiEventContext (which is the context I set per transport address object when I register the event handlers – TdiEventContext is more like a transport address object context) and ConnectionContext (which is the context supplied to TDI in ClientEventConnect() and when I create the connection endpoint with ZwCreateFile). The only exception would be ClientEventConnect() as I get only TdiEventContext but it shouldn’t be a problem because I can have only one connection context object bound to one transport address object that is listening at one time.

Now I see your point and I guess I’ll have to modify some of the existing infrastructure but it should work in the end. But hey at least now I know what I have to do.

Thank you so much for your help!