[RFC] stdbuf: Force stdio's streams initial buffering mode (patch)

Jeremie Le Hen jeremie at le-hen.org
Mon Oct 10 13:23:26 UTC 2011


On Thu, Aug 04, 2011 at 07:19:28PM +0200, Jeremie Le Hen wrote:
> On Sat, Feb 19, 2011 at 07:50:43PM +0100, Jeremie Le Hen wrote:
> > I've been annoyed multiple time when running a command such like
> >     iostat -x 1 | grep -v ad10 | cat -n
> > 
> > The problem stems from two factors:
> >   - grep's stdio sees that its stdout is not a terminal, so stdout is
> >     full buffered and not line-buffered;
> >   - iostat produces output too slowly so the aforementioned buffer takes
> >     numerous seconds to be filled and flushed to the last command.
> > 
> > This problems is not specific to FreeBSD, it is actually a consequence
> > of POSIX specification.  I've checked this on Solaris and Linux.
> > 
> > I've attached a small patch for stdio, so if the environment variable
> > STDIO_IOLBF is set, the output streams will be line-oriented by default.
> >     iostat -x 1 | env STDIO_IOLBF=1 grep -v ad10 | cat -n
> I improved the whole picture.  Now there is a shared library
> libstdbuf.so which can be loaded with LD_PRELOAD and configured through
> a environment variables.  It is able to initial control buffering for
> stdin, stdout and stderr (no buffering, line buffering, block
> buffering).  There is also an utility named stdbuf(1) which can be used
> to run a command with the appropriate environment variables.  Those are
> named after a similar tool in recent versions of GNU coreutils; of
> course, I also borrowed the interface for POLA.
> Here is how to use it (example taken from the manpage):
>     vmstat 1 | stdbuf -o L awk '$2 > 1 || $3 > 1' | cat -n
> I think that using a preloaded shared library is better performance-wise
> because libc doesn't bother looking up configuration variables in the
> environment upon each execve(2), especially for something which is to be
> rarely used.
> Manpages for both stdbuf(1) and libstdbuf(3) are provided.  There is
> surely room for improvement, so feel free to propose corrections to
> them.
> The patch can be applied as-is I think, although I've developped it
> against an old -CURRENT source tree (about 6 months ago).
> I'm looking for some feedback and review and hopefully this feature will
> be committed for FreeBSD 9.0-RELEASE.

It's now too late for 9.0-RELEASE :).  But I'm still willing to see this
feature committed eventually!

While talking about this implementation with a friend, he told me that
having a separate shared library sounds a little bit hackish and two or
three getenv(3) in the libc startup path shouldn't eat too much CPU

Any opinion on this?  Anyone willing to commit this?

Thanks.  Regards,
Jeremie Le Hen

Men are born free and equal.  Later on, they're on their own.
				Jean Yanne

More information about the freebsd-hackers mailing list