Network data transfers to/from user mode

to start, a network bridge. Presumably, with more functionality to come? but what kind? they already communicate via a network protocol, so it can’t be remoting the access. that’s why we are all confused i think

This is part of a larger government contract, so the less I say about “why” the better…

That’s convenient, but remember that a professional engineer has a responsibility to point out when a solution is going the wrong direction.

3rd attempt - who are my edits getting lost??

A “VNIC” that does WHAT, exactly? How do YOU define a VNIC?

VNIC = Virtual Network Adapter

As we so very often say here: “What larger problem are you trying to solve?” Does A send socket traffic to B, which is on a different, pre-defined, IP Address? So, you want this VNIC to fake the IP address?? What is the problem that needs solving here?

This is part of a larger government contract, so the less I say about “why” the better (if we can just leave it at that, it would be good…) :wink:

The requirement (from the powers that be) for the VNIC is to:

  • provide network interface for Software A
  • provide network interface for other future software applications
  • provide ability to bridge disparate network interfaces

**It is important to note that the “lower edge” of the VNIC does not interface directly with h/w; it is the user-mode component PDC ( Software B ) that interfaces with h/w devices that go out to their “cloud”.

Also, for this iteration I wanted to keep Software A & B “as is”.**

I hope this helps to clarify. Sorry for my earlier vagueness. I think in being careful to not divulge too much I just made things more confusing.

Mr @Mark_Holiday … This is frustrating, in many ways,

First, do you see… at the top of the NTDEV list, where it says “ Please Read: Did You Post Something and It Did Not Appear? Edit a Post and Have it Disappear?”? You’ll see this message. Your repeated posts, all 7 of them, were marked as spam. And I had to clean them all up this morning.

And, I’m sorry: “The less said the better” is a cop-out answer, You can be working on a TS/SCI project, and you can describe the engineering requirements in such a way as to not divulge any classified information.

I’m about an inch away from locking this thread, just because it frustrates me and has little chance of providing you any helpful information… mostly because you’re doing nothing to “help us help you.”

You wrote:

The requirement (from the powers that be) for the VNIC is to:

  • provide network interface for Software A
  • provide network interface for other future software applications
  • provide ability to bridge disparate network interfaces

I sincerely fail to understand how Mr. Roberts proverbial “ten line python program” does not adequately fulfill those requirements, if “provide a network interface for Software A” means “provide a connection accessible via sockets.”

Peter

And, you know, I think I probably knew what the initials VNIC stood for. So, thanks for spelling it out for me. /rolls eyes

What I was asking is “What functionality, specifically, do you envision as part of this VNIC” and “what attributes of a NIC, specifically, require emulation”?

Peter

My apologies - and thank you for that well-deserved scolding! :s
Please understand it is certainly not my intention here to create havoc or waste people’s times. I sincerely appreciate the input from everyone here.

I will go back and review what I have for requirements, and try formulate a more lucid request-for-comments here.

Thank you!

VNIC = Virtual Network Adapter

Before we proceed any further, I have to remind you that, assuming that you speak about the full-fledged VNIC that network protocols may bind to, the above mentioned VNIC has to be implemented, on its upper edge, as a NDIS miniport driver. A driver of this kind is not meant to be directly accessed by anyone, apart from NDIS library…

Serving up pending requests (i.e. inverted call model) does seem like the way to go; also,

…which means that it does not receive IRPs from the userland (in fact, from anyone, unless we are speaking about its special control device object that may be used for configuration and has to be the same for all instances of a VNIC, but this is not what we are speaking about here).

Therefore, assuming that you really mean a VNIC, in your particular case you will have to implement it as a layered pair of two kernel modules. Driver A will deal with actually sending and receiving data (if I got it right, this part is done by means of some form of interaction with the userland app), and a driver B will actually implement a VNIC that is presented to the system. It will deal with a driver A on its lower edge and with NDIS library on its upper one, effectively presenting itself a VNIC that protocols may bind to. It is understandable that the inverted call part will be implemented in a driver A.

This is what has to be done in your case if you really mean implementing a VNIC. However, I’ve got a weird feeling that this is a very obvious overkill in your case, and that your problem may be probably solved, indeed, simply by writing a dozen of lines in Python…

Anton Bassov

government contracts resulting in overkill solutions dross and waste? impossible surely!

Thank you for your comments.

Indeed, I was noting that this was part of a government contract project not to “cop out” from providing requirements, but from simply letting you all know that by being sensitive to this, I probably did a poor job in describing the problem at hand.

Also, I am trying to not over-engineer this and make it overly-complicated; this is why I am here in the first place. I am trying to ascertain the best (read: simplest) solution to provide the desired capabilities.

That all being said, I would like to start again, not by describing what we have, but rather what we need.
What we need is a new software “component” that will:

  • interface with 3rd party network applications, e.g. web browsers, at its upper edge, and existing user-mode proprietary protocol/data controller (call it PDC) at its lower edge. (Note: we do own and have full source code for this PDC module)
  • provide the ability to facilitate network bridging at TAP (i.e. Ethernet frame) level. For example, we need to allow a workstation to serve as a “gateway connection”, with multiple PDC modules whose “upper edges” are bridged to a real physical NIC that connects to some other WAN/LAN.

Ideally, this would be the same module. Therefore, one current thought is that a software-only virtual network adapter may solve both of these requirements, similar the TAP-Windows driver used by OpenVPN.

I hope this helps to describe the larger problem(s) that we are trying to solve. If more information is required, please let me know.

One last thing: I am asking for help and suggestions here. If you need to point out that something is not clear, or that more information is needed, I have to ask to please not do so in a condescending manner. Some of you may not realize how some of your comments come across. If that request of mine seems out of line, I apologize, but that is simply my opinion.

Thank you in advance!

So, PROTOCOLs at the upper edge (which are in turn accessed by modules that are accessed by 3rd party network applications)… and Some Special Software (your PDC) at its lower edge. If that meets your needs that an NDIS Virtual NIC driver, with NDIS at the upper edge and KMDF at the lower edge. This is a reasonably vanilla thing to do.

Bridging as you describe it MIGHT be a bit more complicated, it seems to me. You could bridge among PDCs via your NDIS Virtual NIC driver as described above (taking from one PDC and giving to anther PDC)… you’ll have protocol frames after all.

But if your requirement is to bridge these PDCs to “real” LAN/WAN hardware… I’m not sure that’s the same piece of software as we were just describing above (upper edge protocol, lower edge PDC) – Because now you have software that has lower edge that talks to your PDC and also lower edge that talks to a REAL NIC. I’m not sure how you make that work, frankly.

It has been a very, very, long time since I’ve done real work in the NDIS NIC area. All my experience is well before NDIS 6, even. So, I’ll leave the more detailed comments about whether this is one or more than one piece of software to others who are more expert than I am.

We have some really excellent folks from the NDIS team who occasionally stop by here… so mayhaps one of them will dip in to help.

Peter

Thank you, Peter!

The NDIS Virtual NIC driver with KMDF at lower edge was my initial thought here. I have done this before (using the “WDM at lower edge” example as my model), and is probably the reason I was brought on this project. (Like yourself, it has been a few years) :confused:

The reluctance I had with this solution was the requirement that the network traffic at the lower edge actually has to interface with the user-mode PDC, and is the impetus for this whole thread. Employing the “inverted call” method for that data transfer is one possibility that I have learned about here.

But if your requirement is to bridge these PDCs to “real” LAN/WAN hardware… I’m not sure that’s the same piece of software as we were just describing above (upper edge protocol, lower edge PDC) – Because now you have software that has lower edge that talks to your PDC and also lower edge that talks to a REAL NIC. I’m not sure how you make that work, frankly.

I agree that this part of the puzzle requires more investigation.

Employing the “inverted call” method for that data transfer is one possibility that I have learned about here.

So, ignoring the whole “bridge to a real NIC” thing, I wouldn’t hesitate for a second to use Inverted Call … just be sure to have your user-mode PDC post a bunch of pending IOCTLs and handle their completion reasonably efficiently (completion ports are nice for this). It’d be very convenient… you need to send a message from the protocol to the PDC, you pick one of those pending IOCTLs off a manual Queue, copy the NBL to the IOCTL OutBuffer, and WdfRequestCompleteWithImformation and you’re good to go.

Peter

1 Like

But I am still confused as to what you need. These UM components obviously communicate via network protocols now right? It could be TCP or UDP or CDP, but it is some kind of network protocol, or else there would be no use in a virtual NIC - they couldn’t use it unless they use the winsock APIs somehow

Windows has support for bridging interfaces in box since at least Vista IIRC, so just making a bridge driver would be a waste of time

if the IP spaces conflicted, it would be far cheaper to buy a router and program a hairpin NAT than to develop a driver like this

perhaps you want your PDC to stop using a network protocol? In that case, the sockets opened by the other UM processes would have to terminate in your driver, and you could then use inverted call to pass the data up to PDC. KM sockets are well understood and long supported, but the performance difference versus well (or even poorly) coded UM sockets is usually small and they are much more complex to work with (just as everything is harder in KM)

perhaps you want to multi-plex or proxy your PDC in some way? That might be reasonable, but I don’t read that into anything you have said.

I’m intrigued, but confused

1 Like

@MBond2 For the ease of this discussion, we basically want our own virtual network stack, including our own virtual miniport (VNIC).
I have performed this type of work before; one major difference with this project, however, is that the “lower edge” of our miniport does not go out to real hardware or some other kernel component, but rather that traffic interfaces with a user-mode component (PDC), that then talks to the devices for the RF network. For the purposes of this discussion, we are only concerned with facilitating communication between the VNIC and the PDC, i.e. we do not care about the PDC communication with the real RF device(s).

So the impetus for this thread was how to best facilitate that kernel-mode network traffic to/from the user-mode PDC. From this discussion, it appears that the inverted call model provides a solution.

Perhaps this thread got off on a tangent because I did not do an adequate job initially in describing the effort. I hope this helps to clarify.

Mr. Holiday, everybody here already understood that you’ve written a NDIS miniport some time ago and want to build on that.
But probably you’ve learned kernel drivers before normal usermode network (socket) techniques.
Otherwise, why the “PDC” cannot just open a socket on local host and listen, and the client software connect to it?
This would save cost and time of developing a VNIC. On the down side, there will be no driver to write and bill the customer for.

Sorry, you’ve asked to abstain from condescending tone. I’ve failed :frowning:

– pa

@Pavel_A Thank you for your comment. No offense taken with the tone of your comment. :slight_smile:

What you describe is how the system currently operates. For this next project, we eventually would like to move the network interface outside of the PDC component; but we would like to support 3rd party network applications, e.g. web browser. Also, as I mentioned, there is now a requirement to support bridging network interfaces.

Wouldn’t a VNIC/KMDF solution with its “lower edge” talking to the PDC provide/allow for both of these requirements??

If so, then the complexity of this solution - again - becomes:
how to control the flow of network traffic to/from lower edge of VNIC (kernel mode) to/from PDC (in user mode).

That is the problem I originally asked about here; but in trying to ask this is in simple, general terms I must have done a really poor job explaining it because I received many suggestions to just write a user-mode socket application. However, I do think that some folks did manage to figure out what I was trying to ask, as they have suggested the inverted call method to facilitate such communication.

we eventually would like to move the network interface outside of the PDC component; but we would like to support 3rd party network applications, e.g. web browser.

Mark, frankly, the more I read and re-read this thread, the more my confusion grows.
So well, you wish to take some server functionality out of the “PDC” and put it (modified? optimized?) into one or more separate modules.
Why these modules need a separate network interface (VNIC) to talk to the PDC and clients? Why not over the normal sockets?

OTOH if you want to route traffic from Windows apps to the RF network beyond the “PDC”, then yes, a VNIC may be a way to connect that network to the OS. But…

how to control the flow of network traffic to/from lower edge of VNIC (kernel mode) to/from PDC (in user mode).

… your VNIC does not have a lower edge. It has no lower kernel layers and no hardware. It will talk to NDIS at its upper edge - being a normal MAC miniport - and talk to the “PDC” thru a kernel socket, or new ioctl based interface via a helper device object.
How to control the flow of that? Well, it is your project… there are so many sync devices in kernel and user mode…

– pa

… your VNIC does not have a lower edge.

Actually, it does, but this lower edge part is not a physical network - instead, it deals with the userland app/service that provides/consumes data, effectively simulating the physical network…

Anton Bassov

Can this thread just quietly end now? Please?

I think Mr @Mark_Holiday has gotten as much help from us this specific issue as he’s likely to get. If he has additional questions, let him ask in a separate thread.

But we’re going in circles at this point.

Peter

1 Like

Actually, it does, but this lower edge part is not a physical network - instead, it deals with the userland app/service that provides/consumes data, effectively simulating the physical network…

Bingo!

Thank you all for you help on this.