svn commit: r215615 - head/usr.bin/xargs

[Stephen McKay]

On Sunday, 21st November 2010, Stephen McKay wrote:

>Log:
>  xargs can be fooled by exiting children that it did not start, causing
>  it to kick off a new command before the previous has finished, resulting
>  in corrupted (interleaved) output.  It is also fooled by non-exiting
>  children it did not start, failing to exit until all extraneous children
>  have exited.
>  
>  This patch makes xargs keep track of children it starts, ignoring
>  pre-existing ones.

This bug is obscure enough that I think I may need to give some more
detail on how it is triggered.  And the fact it survived 8 years in
a commonly used utility may lead some people to believe it doesn't
exist.

Any program can be exec'd with existing children.  One way this can
happen is a byproduct of the way some shells spawn piped processes.

For example, the following command will result in xargs having find as
a child: zsh -c '(find . -type f | xargs md5)'

Yes, the parentheses are necessary.  Otherwise "find" and "xargs" are
just siblings.

And the following command runs instantly in sh but takes 30s in csh
because the sleep is a child of xargs in the latter and xargs waits
for all children to exit before exiting:
        (sleep 30 & find /COPYRIGHT | xargs ls -l)

When parallel mode (-P) was added to xargs in 2002, there was an
assumption made that every waitpid() call would return a pid that
had been spawned by xargs.  This inadvertently transformed the serial
case into an occasionally parallel case.  If ever an extraneous child
exited during a run, xargs would spawn a 2nd command before the 1st
had finished.  Usually this resulted in output corruption.

Further, it was assumed that waitpid() would always complete in a
timely manner.  This was no longer true because unexpected children
don't have to exit just because xargs' work is done, and xargs was
waiting for all children to exit.

Shells where piped processes never seem to be linked in a parent/child
relationship include sh, bash and ksh.  Shells where it is common
include zsh, tcsh and csh.  Bash seems to be popular, so most people
would never see this problem interactively, and sh is also immune
so no scripts would trigger the bug either.

Since zsh is my favourite shell, I was "lucky" enough to accidentally
(and repeatedly) use command line syntax that triggered this xargs
bug while attempting to verify that some hundreds of gigabytes of
essential data had been successfully copied from a failing mirror
to safer storage during a hairy upgrade.  I had to put a lot of money
in the swear jar, but it hasn't made me give up on xargs!  Or zsh. :-)

Stephen.