kern/177985: [zfs] disk usage problem when copying from one zfs dataset to another on the same pool using mv command

[Bruce Evans]

On Sat, 20 Apr 2013, Andriy Gapon wrote:

> The following reply was made to PR kern/177985; it has been noted by GNATS.
>
> From: Andriy Gapon <avg at FreeBSD.org>
> To: Jon <sybersnake at gmail.com>
> Cc: "bug-followup at FreeBSD.org" <bug-followup at FreeBSD.org>
> Subject: Re: kern/177985: [zfs] disk usage problem when copying from one zfs
> dataset to another on the same pool using mv command
> Date: Sat, 20 Apr 2013 23:25:40 +0300
>
> on 20/04/2013 22:49 Jon said the following:
> ...
> > This all can be
> > accomplished by deleting copied files more often than it currently does or at
> > least adding a flag to turn on synchronized deletes.
>
> No, it can not be accomplished that way, because it would violate how mv(1)
> across filesystems works.  Perhaps it's indeed the time to read the man page?

The behaviour of not removing the source tree until after successful
copying of the whole tree is also specified by POSIX, although a little
unclearly.  I thought that POSIX specified the FreeBSD behaviour (of
using cp -pRP to move across file systems, and then rm -rf of the
source path only if the cp succeeded).  But it seems to have a subtle
different that makes FreeBSD's implementation invalid (apart from the
bugs in cp -pRP):
- for cp -R, it is optional whether cp continues with remaining files (in
   the hierarchy under the current "source file" after a write error.
   FreeBSD's copy does continue.
- for cp -R, the continuing after most other errors (e.g., for open()) is
   not permitted, but FreeBSD's cp continues.
- for mv across file systems, not continuing after write errors seems to
   be required, although a little unclearly.  From POSIX.1-2001-draft7:

@ 25182                       If the duplication of the file hierarchy fails for any reason, mv shall write a diagnostic
@ 25183                       message to standard error, do nothing more with the current source_file, and go on to any
@ 25184                       remaining source_files.

This is fuzzy about when the error for duplication of the file hierarchy
must be detected and acted on, but any reasonable implementation will
detect errors for individual files as they are copied (modulo not detecting
async write errors until much later if at all), and then it takes a
weaselish reading of the above to allow the FreeBSD behaviour of not
quitting as soon as the error is detected, but instead buffering this
error and only reporting it after trying to copy all the other files.

Quitting here is the only way for avoiding falling through to clause 6
which specifies removing the source hierarchy.

@ 25185                       If the duplication of the file characteristics fails for any reason, mv shall write a diagnostic
@ 25186                       message to standard error, but this failure shall not cause mv to modify its exit status.

Continuing in this case is correct.  Not modifying the exit status in this
case is a bug in the spec IMO.  However, FreeBSD's implementation doesn't
have this bug.  For cp -p, the error handling for failing to copy attributes
is underspecified in POSIX (some cases are specified to write a diagnostic on
error, but no cases that I could find are specified to set the exit status
for such errors).  FreeBSD's cp -p sets the exit status for all failures to
copy attributes.  FreeBSD's mv across file system inherits this behaviour.
Thus failure to duplicate a single attribute in a large hierarchy causes
the whole cp to report falure to mv, and mv then consequently modifies its
exit status and refuses to remove the source hierarchy, so it doesn't have
this bug.  In some cases, the attributes might be important and their
might be billions of failures to duplicate them.  Then you don't want the
only copy of them in the source hierarchy being removed.

@ 25187                   6. The file hierarchy rooted in source_file shall be removed. If this fails for any reason, mv shall
@ 25188                       write a diagnostic message to the standard error, do nothing more with the current
@ 25189                       source_file, and go on to any remaining source_files.

We get here for no error in the cp for FreeBSD, or for no error except for
duplicating attributes in POSIX.  Again the spec is fuzzy about when "fails
for any reason" is.  Now it seems best to continue after errors in
intermediate steps, and that is what FreeBSD's rm -rf does.  (We know that
this rm -rf is safer than most, since we have just made a copy successfully.)
POSIX rm -r seems to require continuing after most errors.

Bruce