We’re using named pipes for IPC. As we understand it, named pipes can
withstand process restarts and therefore somebody needs to buffer the
resource, right? Which process gets the memory burden associated with?
To be clear, Process 1 starts a named pipe, and Process 2 joins it.
Process 2 is for all practical purposes only a receiver.
Or is it not in context of a process and therefore doesn’t get
accounted anywhere specific?
Hayden
It is taken care of by the named pipe file system driver, not a process
d
debt from my phone
From: Hayden Livingston
Sent: 7/4/2012 4:34 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Which process does a named pipe resource get committed to?
We’re using named pipes for IPC. As we understand it, named pipes can
withstand process restarts and therefore somebody needs to buffer the
resource, right? Which process gets the memory burden associated with?
To be clear, Process 1 starts a named pipe, and Process 2 joins it.
Process 2 is for all practical purposes only a receiver.
Or is it not in context of a process and therefore doesn’t get
accounted anywhere specific?
Hayden
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
So if Process 1 writes 2GB acorss the named pipe, it’ll get buffered
until Process 2 can pick it up? I’m guessing the theoretical limit is
the physical memory?
Additionally, is this the fastest way to transport data across
processes? I’ve heard people talk about shared memory, but it seems
far more complicated.
On Wed, Jul 4, 2012 at 4:40 PM, Doron Holan wrote:
> It is taken care of by the named pipe file system driver, not a process
>
> d
>
> debt from my phone
> ________________________________
> From: Hayden Livingston
> Sent: 7/4/2012 4:34 PM
> To: Windows System Software Devs Interest List
> Subject: [ntdev] Which process does a named pipe resource get committed to?
>
> We’re using named pipes for IPC. As we understand it, named pipes can
> withstand process restarts and therefore somebody needs to buffer the
> resource, right? Which process gets the memory burden associated with?
>
> To be clear, Process 1 starts a named pipe, and Process 2 joins it.
> Process 2 is for all practical purposes only a receiver.
>
> Or is it not in context of a process and therefore doesn’t get
> accounted anywhere specific?
>
> Hayden
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
The buffering is not infinite, npfs is free to stop you from consuming all va well before you reach that limit. I usually do ipc over and rpc channel. Rpc is free to use pipes, TCP, private os mechanisms etc as the underlying transport without your code worrying about it
d
debt from my phone
From: Hayden Livingston
Sent: 7/4/2012 4:44 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Which process does a named pipe resource get committed to?
So if Process 1 writes 2GB acorss the named pipe, it’ll get buffered
until Process 2 can pick it up? I’m guessing the theoretical limit is
the physical memory?
Additionally, is this the fastest way to transport data across
processes? I’ve heard people talk about shared memory, but it seems
far more complicated.
On Wed, Jul 4, 2012 at 4:40 PM, Doron Holan wrote:
> It is taken care of by the named pipe file system driver, not a process
>
> d
>
> debt from my phone
> ________________________________
> From: Hayden Livingston
> Sent: 7/4/2012 4:34 PM
> To: Windows System Software Devs Interest List
> Subject: [ntdev] Which process does a named pipe resource get committed to?
>
> We’re using named pipes for IPC. As we understand it, named pipes can
> withstand process restarts and therefore somebody needs to buffer the
> resource, right? Which process gets the memory burden associated with?
>
> To be clear, Process 1 starts a named pipe, and Process 2 joins it.
> Process 2 is for all practical purposes only a receiver.
>
> Or is it not in context of a process and therefore doesn’t get
> accounted anywhere specific?
>
> Hayden
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
—
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
Hayden Livingston wrote:
So if Process 1 writes 2GB acorss the named pipe, it’ll get buffered
until Process 2 can pick it up? I’m guessing the theoretical limit is
the physical memory?
When you create the named pipe, you tell it how large the buffers should
be (nInBufferSize and nOutBufferSize). The documentation says those
numbers are only suggestions, but it’s a number way less than physical
memory.
Names pipes are very, very similar to sockets. PIPE_TYPE_BYTE is
essentially TCP, and PIPE_TYPE_MESSAGE is essentially UDP.
Additionally, is this the fastest way to transport data across
processes? I’ve heard people talk about shared memory, but it seems
far more complicated.
The only additional complication is setting up the pointer management.
For example, it’s possible to set up a circular buffer in shared memory,
so that each end only updates its own pointer. That can be done safely
with a minimum of locking.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
“Hayden Livingston” wrote in message
news:xxxxx@ntdev…
> Additionally, is this the fastest way to transport data across
> processes? I’ve heard people talk about shared memory, but it seems
> far more complicated.
>
Memory mapped files should be the best supported IPC solution performance
wise.
Shared resident memory needs backing of a kernel driver which allocates,
locks and maps the memory in the processes sharing the memory. It also
requires keeping track of the lifetime of the processes in which the memory
is mapped. So yes, this is somewhat complicated and easy to get wrong. But
it does allow you to avoid hard pagefaults if latency is a concern.
//Daniel
Fastest? What performance criterion are you specifically concerned about?
Or is this just a ‘I want to do this in an efficient way’ statement?
Assuming the latter, your choice of language will probably define the least
work for you solution that is reasonably efficient. In C++ RPC is not too
bad, but in C message oriented pipes are much simpler and in C# there are
many other possibilities.
Assuming the former, you need to understand what the performance dynamics of
your particular problem are before anyone here can tell you what the
‘fastest’ anything is because, as usual, the answer is ‘it depends’.
“Hayden Livingston” wrote in message news:xxxxx@ntdev…
So if Process 1 writes 2GB acorss the named pipe, it’ll get buffered
until Process 2 can pick it up? I’m guessing the theoretical limit is
the physical memory?
Additionally, is this the fastest way to transport data across
processes? I’ve heard people talk about shared memory, but it seems
far more complicated.
On Wed, Jul 4, 2012 at 4:40 PM, Doron Holan
wrote:
> It is taken care of by the named pipe file system driver, not a process
>
> d
>
> debt from my phone
> ________________________________
> From: Hayden Livingston
> Sent: 7/4/2012 4:34 PM
> To: Windows System Software Devs Interest List
> Subject: [ntdev] Which process does a named pipe resource get committed
> to?
>
> We’re using named pipes for IPC. As we understand it, named pipes can
> withstand process restarts and therefore somebody needs to buffer the
> resource, right? Which process gets the memory burden associated with?
>
> To be clear, Process 1 starts a named pipe, and Process 2 joins it.
> Process 2 is for all practical purposes only a receiver.
>
> Or is it not in context of a process and therefore doesn’t get
> accounted anywhere specific?
>
> Hayden
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
Fair enough.
The person sending the data over the pipe is actually a C# program. I
don’t think that should matter specifically, but there you go.
I don’t care about latency specifically, but data rate is pretty high
around 10MB/sec … so it’s more about throughput.
I’m going to go with named pipes, cause they are dead simple.
Hayden.
On Thu, Jul 5, 2012 at 3:25 PM, m wrote:
> Fastest? What performance criterion are you specifically concerned about?
> Or is this just a ‘I want to do this in an efficient way’ statement?
>
> Assuming the latter, your choice of language will probably define the least
> work for you solution that is reasonably efficient. In C++ RPC is not too
> bad, but in C message oriented pipes are much simpler and in C# there are
> many other possibilities.
>
> Assuming the former, you need to understand what the performance dynamics of
> your particular problem are before anyone here can tell you what the
> ‘fastest’ anything is because, as usual, the answer is ‘it depends’.
>
> “Hayden Livingston” wrote in message news:xxxxx@ntdev…
>
>
> So if Process 1 writes 2GB acorss the named pipe, it’ll get buffered
> until Process 2 can pick it up? I’m guessing the theoretical limit is
> the physical memory?
>
> Additionally, is this the fastest way to transport data across
> processes? I’ve heard people talk about shared memory, but it seems
> far more complicated.
>
> On Wed, Jul 4, 2012 at 4:40 PM, Doron Holan
> wrote:
>>
>> It is taken care of by the named pipe file system driver, not a process
>>
>> d
>>
>> debt from my phone
>> ________________________________
>> From: Hayden Livingston
>> Sent: 7/4/2012 4:34 PM
>> To: Windows System Software Devs Interest List
>> Subject: [ntdev] Which process does a named pipe resource get committed
>> to?
>>
>> We’re using named pipes for IPC. As we understand it, named pipes can
>> withstand process restarts and therefore somebody needs to buffer the
>> resource, right? Which process gets the memory burden associated with?
>>
>> To be clear, Process 1 starts a named pipe, and Process 2 joins it.
>> Process 2 is for all practical purposes only a receiver.
>>
>> Or is it not in context of a process and therefore doesn’t get
>> accounted anywhere specific?
>>
>> Hayden
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
By the person, I meant process.
On Thu, Jul 5, 2012 at 3:47 PM, Hayden Livingston
wrote:
> Fair enough.
>
> The person sending the data over the pipe is actually a C# program. I
> don’t think that should matter specifically, but there you go.
>
> I don’t care about latency specifically, but data rate is pretty high
> around 10MB/sec … so it’s more about throughput.
>
> I’m going to go with named pipes, cause they are dead simple.
>
> Hayden.
>
> On Thu, Jul 5, 2012 at 3:25 PM, m wrote:
>> Fastest? What performance criterion are you specifically concerned about?
>> Or is this just a ‘I want to do this in an efficient way’ statement?
>>
>> Assuming the latter, your choice of language will probably define the least
>> work for you solution that is reasonably efficient. In C++ RPC is not too
>> bad, but in C message oriented pipes are much simpler and in C# there are
>> many other possibilities.
>>
>> Assuming the former, you need to understand what the performance dynamics of
>> your particular problem are before anyone here can tell you what the
>> ‘fastest’ anything is because, as usual, the answer is ‘it depends’.
>>
>> “Hayden Livingston” wrote in message news:xxxxx@ntdev…
>>
>>
>> So if Process 1 writes 2GB acorss the named pipe, it’ll get buffered
>> until Process 2 can pick it up? I’m guessing the theoretical limit is
>> the physical memory?
>>
>> Additionally, is this the fastest way to transport data across
>> processes? I’ve heard people talk about shared memory, but it seems
>> far more complicated.
>>
>> On Wed, Jul 4, 2012 at 4:40 PM, Doron Holan
>> wrote:
>>>
>>> It is taken care of by the named pipe file system driver, not a process
>>>
>>> d
>>>
>>> debt from my phone
>>> ________________________________
>>> From: Hayden Livingston
>>> Sent: 7/4/2012 4:34 PM
>>> To: Windows System Software Devs Interest List
>>> Subject: [ntdev] Which process does a named pipe resource get committed
>>> to?
>>>
>>> We’re using named pipes for IPC. As we understand it, named pipes can
>>> withstand process restarts and therefore somebody needs to buffer the
>>> resource, right? Which process gets the memory burden associated with?
>>>
>>> To be clear, Process 1 starts a named pipe, and Process 2 joins it.
>>> Process 2 is for all practical purposes only a receiver.
>>>
>>> Or is it not in context of a process and therefore doesn’t get
>>> accounted anywhere specific?
>>>
>>> Hayden
>>>
>>> —
>>> NTDEV is sponsored by OSR
>>>
>>> For our schedule of WDF, WDM, debugging and other seminars visit:
>>> http://www.osr.com/seminars
>>>
>>> To unsubscribe, visit the List Server section of OSR Online at
>>> http://www.osronline.com/page.cfm?name=ListServer
>>>
>>>
>>> —
>>> NTDEV is sponsored by OSR
>>>
>>> For our schedule of WDF, WDM, debugging and other seminars visit:
>>> http://www.osr.com/seminars
>>>
>>> To unsubscribe, visit the List Server section of OSR Online at
>>> http://www.osronline.com/page.cfm?name=ListServer
>>
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
10 MB/s should not be a problem for reasonable hardware using named pipes
locally. You may start to notice a delay if you get to ~100MB/s or so
depending on your chipset etc.
“Hayden Livingston” wrote in message news:xxxxx@ntdev…
By the person, I meant process.
On Thu, Jul 5, 2012 at 3:47 PM, Hayden Livingston
wrote:
> Fair enough.
>
> The person sending the data over the pipe is actually a C# program. I
> don’t think that should matter specifically, but there you go.
>
> I don’t care about latency specifically, but data rate is pretty high
> around 10MB/sec … so it’s more about throughput.
>
> I’m going to go with named pipes, cause they are dead simple.
>
> Hayden.
>
> On Thu, Jul 5, 2012 at 3:25 PM, m wrote:
>> Fastest? What performance criterion are you specifically concerned
>> about?
>> Or is this just a ‘I want to do this in an efficient way’ statement?
>>
>> Assuming the latter, your choice of language will probably define the
>> least
>> work for you solution that is reasonably efficient. In C++ RPC is not
>> too
>> bad, but in C message oriented pipes are much simpler and in C# there are
>> many other possibilities.
>>
>> Assuming the former, you need to understand what the performance dynamics
>> of
>> your particular problem are before anyone here can tell you what the
>> ‘fastest’ anything is because, as usual, the answer is ‘it depends’.
>>
>> “Hayden Livingston” wrote in message news:xxxxx@ntdev…
>>
>>
>> So if Process 1 writes 2GB acorss the named pipe, it’ll get buffered
>> until Process 2 can pick it up? I’m guessing the theoretical limit is
>> the physical memory?
>>
>> Additionally, is this the fastest way to transport data across
>> processes? I’ve heard people talk about shared memory, but it seems
>> far more complicated.
>>
>> On Wed, Jul 4, 2012 at 4:40 PM, Doron Holan
>> wrote:
>>>
>>> It is taken care of by the named pipe file system driver, not a process
>>>
>>> d
>>>
>>> debt from my phone
>>> ________________________________
>>> From: Hayden Livingston
>>> Sent: 7/4/2012 4:34 PM
>>> To: Windows System Software Devs Interest List
>>> Subject: [ntdev] Which process does a named pipe resource get committed
>>> to?
>>>
>>> We’re using named pipes for IPC. As we understand it, named pipes can
>>> withstand process restarts and therefore somebody needs to buffer the
>>> resource, right? Which process gets the memory burden associated with?
>>>
>>> To be clear, Process 1 starts a named pipe, and Process 2 joins it.
>>> Process 2 is for all practical purposes only a receiver.
>>>
>>> Or is it not in context of a process and therefore doesn’t get
>>> accounted anywhere specific?
>>>
>>> Hayden
>>>
>>> —
>>> NTDEV is sponsored by OSR
>>>
>>> For our schedule of WDF, WDM, debugging and other seminars visit:
>>> http://www.osr.com/seminars
>>>
>>> To unsubscribe, visit the List Server section of OSR Online at
>>> http://www.osronline.com/page.cfm?name=ListServer
>>>
>>>
>>> —
>>> NTDEV is sponsored by OSR
>>>
>>> For our schedule of WDF, WDM, debugging and other seminars visit:
>>> http://www.osr.com/seminars
>>>
>>> To unsubscribe, visit the List Server section of OSR Online at
>>> http://www.osronline.com/page.cfm?name=ListServer
>>
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
> I don’t care about latency specifically, but data rate is pretty high
around 10MB/sec
This is pretty low. I would not consider performance issues with this.
C language and Win32 API as an ecosystem have no good ways of doing IPC. You can use named pipes, but you will need to do the marshalling yourself, which is a dirty kind of work.
You can use C#/Java solutions, but then you will have JNI or p/Invoke, which is another pile of dirt (and yes, marshalling again).
I would seriously consider C++/DCOM. It has no issues calling C-style Win32, unlike C#, and the IPC stuff there is, though worse and dirtier then .NET’s one, is much better then coding all marshalling stuff yourself.
And no, Windows named pipes (unlike UNIX ones) do not survive process restart, they are volatile in memory objects which are destroyed when both processes close both file handles.
–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com