SOLVED, BUT UGLY, was Re: rsync update mixed environment

[David Benfell]

Thanks Matthew for your response. My specific responses to your
questions and suggestions are below. For everybody else, you probably
don't want to wade through all that; I'm top-posting the TL;DR version here.

Basically, I couldn't get rsync to update from within a script. No idea
why but it looks like some way, somehow the script was the problem.
Realizing it would be completely unreasonable to expect people on this
list to debug my fairly gnarly script, I gave up and am using a new
script to find the local and (with ssh) remote modification times. In
the script that wasn't working, I'm using this new script to obtain
those times, compare them, and scp accordingly. This works like a charm.

Yes, I'm wondering why I can go to all that trouble, get it working, and
utterly fail to get rsync working.

My script to obtain the modification time is at
https://parts-unknown.org/stikked/view/2e00aa88 but be sure to read the
comments because using stat turns out to be really, really, really ugly
and weird, which means, among other things, you have to use it with
/dev/null on STDIN (this seems to be true on both Linux and FreeBSD).
Again, no idea why. I'm in debt to a couple stackoverflow posts here.

On 04/12/2016 12:49 AM, Matthew Seaman wrote:
> On 12/04/2016 08:04, David Benfell wrote:
>> I'm used to rsync from Linux, where one can do rsync -auvz to update the
>> files from one directory to another between networked systems.
>>
>> The z option gave me trouble in FreeBSD but I guess you can do zz
>> instead. I guess I missed a memo on whatever is going on with this.
> The FreeBSD port is using the zlib from the base system rather than what
> is bundled with rsync.  This does have an effect on functionality,
> particularly on interoperability with other versions of rsync.  If
> you're using rsync in a diverse environment, it might be an idea to
> toggle that option setting in the rsync port and reinstall it.
I found and disabled the option. It should now be using the bundled
zlib. However, rsync from Linux to FreeBSD still fails.

But testing further, the failure only occurs within a script (there are
a number of files and directories I'm trying to keep in sync) in which I
first try to sync from the FreeBSD system to the Linux system. This must
be making a difference, but I'm not understanding why, have no idea what
questions to ask about why it might be making a difference, and
reversing the order of the operations makes no difference.

More generally, it's even failing to create files on the (FreeBSD)
destination that do not previously exist.

An obvious but both rude and inconvenient solution to this might be to
elide the u option or to go to something simpler like scp (from ssh) but
we're talking lots and lots of gigabytes, sometimes to and from a
notebook that might be operating remotely from a coffee shop. I found
unison, but it's now fussy about versions so a mixed environment won't
work at all.
>> Either way, updates don't seem to happen reliably when I do this from a
>> Linux system to a FreeBSD system. The failure is silent; it just doesn't
>> seem to recognize that a file in the source is newer than the version in
>> the destination. And then I wonder why an updated file didn't propagate.
> Curious.  Is there any problem going in the other direction?
A test going the other direction succeeded.
>   FreeBSD to
> Linux?  What are the versions of rsync on either side?
At first blush, the versions are the same. But there are some
differences in capabilities and I just don't know enough to even guess
whether they're significant:

On Linux:

rsync --version
rsync  version 3.1.2  protocol version 31
<snip>
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc
<snip>

On FreeBSD (both with and without bundled zlib):

rsync --version
rsync  version 3.1.2  protocol version 31
<snip>
Capabilities:
    64-bit files, 32-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, no prealloc, file-flags
<snip>
>   Does using
> compression or not affect the reliability?  (ie. try your rsync without
> any compression at all.)
Yes, I have tried it without compression. That's actually the condition
I initially encountered the problem under because my initial reaction to
the error message about the z option was simply to elide it--these
transfers are mostly occurring on a local network so speed is a bit less
of a concern.
>   I'm thinking this could be a problem due to
> the change in the way the compression option works with different
> rsync(1) versions.  Quoting the man page:
>
>                                          This  matching-data  com-
>   pression  comes at a cost of CPU, though, and can be disabled by
>   repeating the -z option, but only if both  sides  are  at  least
>   version 3.1.1.

-- 
David Benfell, Ph.D.
benfell at parts-unknown.org


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20160412/499bd41e/attachment.sig>