Re: Odd behaviour of two identical ZFS servers mirroring via rsync

On Thu, 17 Nov 2022, andy thomas wrote:

> On Thu, 17 Nov 2022, Freddie Cash wrote:
> 
> > Now that you have it working with rsync, you should look into using ZFS
> > send/recv as an alternative. You should find it finishes a lot quicker than
> > rsync, although it does require a bit more scripting know-how (especially if
> > you want to use restartable/interruptible transfers, or use a transport
> > other than SSH for better throughout).
> > ZFS send/recv works "below" the filesystem later today rsync works at. ZFS
> > knows which individual blocks on disk have changed between snapshots and
> > only transfers those blocks. There's no file comparisons and hash
> > computations to work out between the hosts.
> > 
> > Transferring the initial snapshot takes a long time, though, as it has to
> > transfer the entire filesystem across. Transferring individual snapshots
> > after that takes very little time. It's similar to doing a "full" backup,
> > and then "incrementals".
> > 
> > When transferring data between ZFS pools with similar filesystem
> > hierarchies, you really should consider send/recv.
> 
> Point taken! Three days ago, one of our HPC users who has ~9TB of data stored
> on our server decided to rename a subdirectory containing ~4TB of experimental
> data stored as many millions of relatively small files within a lot of
> subdirectories. As a result, rsync on the destination (mirror) server is still
> deleting his old folder and its contents and hasn't even started mirroring the
> renamed folder.
> 
> Since our servers have been up for 5.5 years and are both well overdue for an
> O/S upgrade from FBSD 11.3 to 13.x anyway, I think this would be a good
> opportunity to switch from rsync to ZFS send/recv. I was planning to do the
> O/S update over the upcoming Christmas vacation when HPC demand here
> traditionally falls to a very low level - I will set up a pair of test servers
> in the next day or two, play around with this and get some experience of this
> before upgrading the 'live' servers.

[snip]

you may look at zxfer port as a start point of ZFS send/recv scripting 

excerpt from our backup script:

	SSHOPT="-i /home/backup/.ssh/id_ed25519 backup@$MACHINE"
        zxfer -FkPv -U -o compression=lz4 \
                -O "$SSHOPT" \
                -R $POOL ${BASEFS}/zfs/$MACHINE

(initiated from backup machine to pull snapshots missing for the moment; 
target servers use zfsnap2 port daily with 6d/5w/3m retention for d/w/m 
snapshots respectively)

-- 
Sincerely,
D.Marck                                        [DM5020, MCK-RIPE, DM3-RIPN]
[ FreeBSD committer:                                    marck@FreeBSD.org ]
---------------------------------------------------------------------------
*** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- woozle@woozle.net ***
---------------------------------------------------------------------------