bin/144411: [patch] mtree(8) doesn't reject non-regular files for -X

[Bruce Evans]

On Tue, 30 Mar 2010, Garrett Cooper wrote:

> Hi,
>     I'm not 100% satisfied with this patch now. Looking back it fails
> the following case:
>
>      -P    Do not follow symbolic links in the file hierarchy, instead con-
>            sider the symbolic link itself in any comparisons.  This is the
>            default.

-P should have the same semantics and description in all utilities.  The
description should not have grammar errors like the above (comma splice).
In cp(1) the description is:

%%%
      -P    If the -R option is specified, no symbolic links are followed.
            This is the default.
      ...
      Symbolic links are always followed unless the -R flag is set, in which
      case symbolic links are not followed, by default.  The -H or -L flags (in
      conjunction with the -R flag) cause symbolic links to be followed as
      described above.  The -H, -L and -P options are ignored unless the -R
      option is specified.  In addition, these options override each other and
      the command's actions are determined by the last one specified.
%%%

I now see that the grammar error is from the original version of mtree(1),
and is probably one of the things you don't like.  mtree also has -L, but
not -R or -P or -h.  It is not clear how any utility that traverses trees
can work without a full complement of -[HLPR] or how any utility that
supports symlinks can work without -h.  cp(1) doesn't support -h either,
but I think mtree is more like chmod(1) here -- it supports setting
attributes (without having an source file or symlink in the hierarchy to
copy the attributes from like cp does), so it should have a way control
whether the attributes are set on a symlink or on the symlink target.

>     I need to add a check for this option to the patch, use stat if
> the `follow links' state is true, otherwise use lstat.

In mtree, -P controls only the tree traversal.  If symlinks are not followed
in the traversal, then it can't be right to always follow them for the final
path component like the current code always does.  The current code never
use, lstat(), lchmod(), lutimes(), etc., so it can't handle symlinks very
well.  You would need to change much more than stat() to lstat() to fix this.
First, there are delicate combinations of the -[HLPR] flags to distinguish
between following symbolic links on the command line and symbolic links in
the file hierarchy.  In cp, these are:

     -RH: follow on the command line but not in the hierarchy
     -RL: always follow
     -RP: never follow.

mtree has -R as implicit and doesn't have -H.  It claims that -P is the
default.  However, this cannot possibly work for leaf symlinks, since
mtree never uses lstat(), lchmod(), etc., so it always follows leaf
symlinks.  Next, there is the -h flag to give control over following
leaf symlinks.  In cp, this isn't really needed since it makes no sense
to use a different "follow" rule for the source and target; so with -R,
-P actually gives "never follow", while without -R, -P has no effect
and gives "always follow"; thus you can get the effect of -h using -RP
for copying a single symlink, and the only difficult operation not getting
the effect of -h for copying hierachies.  In mtree, there is no -h flag,
and no support for it either, and the claimed "never follow" for -P is
just wrong for leaf symlinks.

Oops, further grepping shows that the above is not quite right.  mtree
does use lchown() for creating symlinks.  This might be because ownership
is more important for security than other attributes.  Also, the claim
that -RP always follows symlinks seems to be wrong in all cases except
in cp: in chflags, chmod and chown, it takes -h to follow leaf symlinks;
cp doesn't have -h and -RP is required to preserve everything, so cp
is more careful about this and uses lch*() on symlinks (even for cp, it
took about 10 years for it to catch up with the kernel and actually use
the newer lch*() syscalls).

chmod(1) and chown(1) and their history are probably better refences
than cp(1) for deciding what mtree should do.  In 4.4BSD, chown and
chmod had all of -HLPR, but not -h, and chmod(1) and chown(8) on a symlink
were documented as having no effect; since chmod(2) and chown(2) do have
an affect (they follow the symlink), this must have been implemented by
not calling these syscalls on a symlink at all, but this seems to have
been slightly broken (the call was not always avoided as documented).
FreeBSD added -h and eventually the newer lch*() calls, but doesn't
seem to have changed the avoidance of the call, so the mismatch between
the documentation and the behaviour for chown(1) now is:

%      The chown utility changes the user ID and/or the group ID of the speci-
%      fied files.  Symbolic links named by arguments are silently left
%      unchanged unless -h is used.

Actually:

     Symbolic links named by arguments are followed unless -h (or certain
     other options) are used.

% 
%      The options are as follows:
% 
%      -H      If the -R option is specified, symbolic links on the command line
%              are followed.  (Symbolic links encountered in the tree traversal
%              are not followed.)
% 
%      -L      If the -R option is specified, all symbolic links are followed.

Not checked.

% 
%      -P      If the -R option is specified, no symbolic links are followed.
%              This is the default.

Actually:

        -P alone:
        This is the default.  Contrary to the above (specifically for -P and
        generally with no args), all symbolic links are followed, since
        there is no tree traversal so we just have pathnames on the
        command line and we apply chown(2) to these.

        -Ph:
        No symbolic links are followed.  lchown() is applied to everything
        on the command line, thus changing the links themselves for symbolic
        links on the command line.

        -P with -R:
        No symbolic links are followed.  Moreover, for symbolic links
        encountered in the traversal, no attempt is made to change the
        ownership of either the link or its target.  This is _not_ the
        default!  [I only checked this for leaf symbolic links to regular
        files.]

        -P with -R and -h:
        No symbolic links are followed.  For leaf symbolic links encountered
        in the traversal, change the ownership of the link itself.

% 
%      -R      Change the user ID and/or the group ID of the specified directory
%              trees (recursively, including their contents) and files.  Beware
%              of unintentionally matching the ``..'' hard link to the parent
%              directory when using wildcards like ``.*''.

Actually:

        Presumably mostly works, but it doesn't say anything about the non-
        change or change to symlinks (see new descriptions above), and
        it doesn't say anything about the non-recursion or recursion into
        symlinks (see old descriptions under -H and -L -- normally the
        recursion stops at a symlink; with -L here can only be leaf symlinks
        to consider changing and there good possibilities for confusing
        differences between -L[h] and -P[h] on the leaves that are reached
        by both).

% ...
%      -h      If the file is a symbolic link, change the user ID and/or the
%              group ID of the link itself.

Actually:

        Presumably mostly works.  Not checked except in combination with -P
        as above.

The confusing cases seem to be just bugs.  Even old POSIX (2001 draft
7) seems to be missing these bugs.  E.g., it requires chown -RP to
change the symlink itself if the system supports chown(2) on symlinks,
and it specifies -h for some systems.  The FreeBSD behaviour seems to
be a vestige of the 4.4BSD mistake of not supporting chown(2) on
symlinks -- with this mistake, not attempting to change anything made
no difference.

The above was tested under FreeBSD-~5.2.  -current is a little different:
- I only tested chmod(1) since I don't have permission on any available
   system to use chown(2)
- chmod(1) doesn't permit -h in combination with -R (even under ~5.2) so
   I couldn't do complete tests even for chmod
- -RP still fails to make any attempt to change anything.
- Old POSIX is deficient for chmod(1).  It only specifies a -R option.
   My network is too slow today to check what current POSIX specifies.
   (Current POSIX encourages full attributes for symlinks so it presumably
   has to specify -[HLRPh] for all tree-traversing and/or symlink-supporting
   utilities.)

Sorry this is so long.  I only knew that -[HLRPh] was complicated when
I started.

Bruce