Asynchronous pipe I/O

rihad rihad at
Wed Nov 5 04:23:09 PST 2008

Imagine this shell pipeline:

sh prog1 | sh prog2

As given above, prog1 blocks if prog2 hasn't yet read previously written 
data (actually, newline separated commands) or is busy. What I want is 
for prog1 to never block:

sh prog1 | buffer | sh prog2

I first thought that the aptly named misc/buffer port would do exactly 
what I wanted: buffering prog1 output for prog2 to read it at its 
earliest convenience. That way prog1 would never block (unless it hit 
buffer's memory limits). Alas, misc/buffer was originally designed for 
tape backups, and despite its author's stating its applicability to 
other uses:

"This is a program designed initially to speed up writing tapes on 
remote tape drives, but may be used as a general pipe buffering utility."

buffer never starts writing unless the limit given by -s is crossed, 
which is 10 kbytes by default and cannot be less than 496 bytes, which 
is too much for me. Ideally I want it to start writing immediately, 
whenever new data hits its pools. Unfortunately, the -p 0 option doesn't 
work either:

        -s size
             Size in bytes of each block.  The  default  blocksize  is
             10k  to match the normal output of the tar(1) program.

       -p percentage
             Only start a write when the given percentage of the internal
             queue is  full.   A  percentage around 75 often proves best.
             Defaults to zero.

Wouldn't such an intermediary tool be a great way to boost performance 
for certain types of solutions?

Thanks for any tips (Sorry if this was an inappropriate place to ask)

More information about the freebsd-questions mailing list