Asynchronous pipe I/O

rihad rihad at mail.ru
Wed Nov 5 08:30:37 PST 2008


Imagine this shell pipeline:

sh prog1 | sh prog2


As given above, prog1 blocks if prog2 hasn't yet read previously written
data (actually, newline separated commands) or is busy. What I want is
for prog1 to never block:

sh prog1 | buffer | sh prog2

I first thought that the aptly named misc/buffer port would do exactly
what I wanted: buffering prog1 output for prog2 to read it at its
earliest convenience. That way prog1 would never block (unless it hit
buffer's memory limits). Alas, misc/buffer was originally designed for
tape backups, and despite its author's stating its applicability to
other uses:

"This is a program designed initially to speed up writing tapes on
remote tape drives, but may be used as a general pipe buffering utility."


buffer never starts writing unless the limit given by -s is crossed,
which is 10 kbytes by default and cannot be less than 496 bytes, which
is too much for me. Ideally I want it to start writing immediately,
whenever new data hits its pools. Unfortunately, the -p 0 option doesn't
work either:

        -s size
             Size in bytes of each block.  The  default  blocksize  is
             10k  to match the normal output of the tar(1) program.

       -p percentage
             Only start a write when the given percentage of the internal
             queue is  full.   A  percentage around 75 often proves best.
             Defaults to zero.

Wouldn't such an intermediary tool be a great way to boost performance
for certain types of solutions?

Thanks for any tips (Sorry if this was an inappropriate place to ask)



More information about the freebsd-hackers mailing list