Asynchronous pipe I/O

[rihad]

Peter Jeremy wrote:
> On 2008-Nov-05 17:40:11 +0400, rihad <rihad at mail.ru> wrote:
>> Imagine this shell pipeline:
>>
>> sh prog1 | sh prog2
>>
>>
>> As given above, prog1 blocks if prog2 hasn't yet read previously written
>> data (actually, newline separated commands) or is busy. What I want is
>> for prog1 to never block:
>>
>> sh prog1 | buffer | sh prog2
> 
> There's also misc/mbuffer which is supposed to be an enhancement of
> misc/buffer - though I haven't used either.  I have a program I wrote
> to do this but it's not in a releasable state.
> 
That thing only works for certain types of devices (tape?):
$ while :; do echo hi; sleep 1; done | mbuffer -q | cat
warning: Could not stat output device (unsupported by system)!
This can result in incorrect written data when
using multiple volumes. Continue at your own risk!
^C
$

While writing this email I tried another incantation, and buffer finally 
worked!

$ while :; do echo hi; sleep 1; done | buffer -s1 -b100 | cat
hi
hi
hi
hi
hi
^C
(each line was output one second apart).

Once again thank to everybody participating!

>> Wouldn't such an intermediary tool be a great way to boost performance
>> for certain types of solutions?
> 
> I've found that for dump|restore or dump|gzip, I can get quite significant
> speedups by adding a buffer that is several hundred MB in the middle.
> 
Well, if OS buffers aren't good enough, then throwing some memory at 
disk I/O surely helps ;)