From nobody Thu Nov 17 09:31:02 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NCZRS01mqz4hfQ0 for ; Thu, 17 Nov 2022 09:31:16 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NCZRR4rM4z4CN1 for ; Thu, 17 Nov 2022 09:31:15 +0000 (UTC) (envelope-from fjwcash@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-wm1-x330.google.com with SMTP id j5-20020a05600c410500b003cfa9c0ea76so1358841wmi.3 for ; Thu, 17 Nov 2022 01:31:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Ld2W/X3ruLbjZPKorS4VZSJBhyOs5fZvq6zDK+tgX/g=; b=o1DzoQp/h5gUV3SRydfdMBgcpzA7X1+vlONLep1U7jUvjRJ17ZmLqpqUwIXXS4Oj4W MpR8XT6xDfIssmCshkJEIzaDs6pT/TMonrlfww6VZpgaTcZ1muN8y5p6SmU3GGRvV3LQ WjJOQTDl2xmWAMnAOIvNIA9iwz++vXr4rrO/W7r9KahHH2pRsSKqT4mBajlb1xBfTMiX SLtnssuyoP8a6xnu20H0c4MwGs5+KRNvjFlOe/XSFTO/zhbn5IIATY6SVklqlHmyUr2E Tk/+zAvhw1lDnkcmEkaxDFvoo1wEq5Wa0Dj+/rap42RXyVSeDPwXrw2AOyh+Fe+Rjs1m IR1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Ld2W/X3ruLbjZPKorS4VZSJBhyOs5fZvq6zDK+tgX/g=; b=jSRY2/ElO+7J7vRB4GyKvNWOCcF0AHQzj6MIY88vFDjyXR10KV/A/HLgQ46Amurkqq eldXdVTBxTyj8Bfm3Kskwm/7bvkNh+UGObJwIyusaVrlouimdWV+6J7If4G1YmJFVUD/ qxIrRgjIecYIqJFo76b6AL0tRXGNWqQ688LQIHi0yKf8kEaHD8rKc818ugMsGlUg2zyD F3MfrcVi1pZx5txdKEhyCE9h827YmOBNJki0nCRPQ2MSemx+t23uinOit3VMV19WKoJ1 epYwLPjidyZOsxmIhZe4wqISwJySUzaLADRIlGR201F1pJ6EqvrecXIK5bkY6ZoT4Q54 vUBQ== X-Gm-Message-State: ANoB5pkKVBtp0nYBhAxQKkYRKqdIi0qclfj4CKYuTJqp3Zvzes1sXR1b 8EgmeWq6v/YAwDaTRpdebsL8GPQBUTzOIKMB5hk= X-Google-Smtp-Source: AA0mqf7ZTH7WkvDH1Y7EgyVhhcrd2VYMGe2TPBoFHkTY1HRcAqaIKUIaV/3gOIFb78JSorCpWDWEv0tMn5V7swwjnik= X-Received: by 2002:a05:600c:511c:b0:3cf:6c05:809e with SMTP id o28-20020a05600c511c00b003cf6c05809emr1103350wms.74.1668677473755; Thu, 17 Nov 2022 01:31:13 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Freddie Cash Date: Thu, 17 Nov 2022 01:31:02 -0800 Message-ID: Subject: Re: Odd behaviour of two identical ZFS servers mirroring via rsync To: andy thomas Cc: Bob Friesenhahn , Mark Saad , FreeBSD Filesystems Content-Type: multipart/alternative; boundary="00000000000011651005eda74014" X-Rspamd-Queue-Id: 4NCZRR4rM4z4CN1 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --00000000000011651005eda74014 Content-Type: text/plain; charset="UTF-8" Now that you have it working with rsync, you should look into using ZFS send/recv as an alternative. You should find it finishes a lot quicker than rsync, although it does require a bit more scripting know-how (especially if you want to use restartable/interruptible transfers, or use a transport other than SSH for better throughout). ZFS send/recv works "below" the filesystem later today rsync works at. ZFS knows which individual blocks on disk have changed between snapshots and only transfers those blocks. There's no file comparisons and hash computations to work out between the hosts. Transferring the initial snapshot takes a long time, though, as it has to transfer the entire filesystem across. Transferring individual snapshots after that takes very little time. It's similar to doing a "full" backup, and then "incrementals". When transferring data between ZFS pools with similar filesystem hierarchies, you really should consider send/recv. Cheers, Freddie Typos due to smartphone keyboard. On Thu., Nov. 17, 2022, 12:50 a.m. andy thomas, wrote: > I thought I would report back that changed my rsync options from '-Wav > --delete' to '-av --inplace --no-whole-file --delete' has made a > significant difference, with mirrored directory sizes on the slave server > now falling and approaching the original sizes on the master. The only > downside is that since whole-file replication is obviously a lot faster > than updating the changed parts of individual files, mirroring is now > taking longer than 24 hours so this will be changed to every few days or > even weekly when more is known about user behaviour on the master server. > > Andy > > On Sun, 13 Nov 2022, Bob Friesenhahn wrote: > > > On Sun, 13 Nov 2022, Mark Saad wrote: > >>> > >> Bob are you saying when the target is zfs --inplace --no-whole-file > helps > >> or just in general when you have > >> large files ? Also have you tried using --delete-during / > --delete-after > >> ? > > > > The '-inplace --no-whole-file' updates the file blocks if they have > changed > > (comparing the orgin blocks with the existing mirror blocks) rather than > > creating a new copy of the file and moving it into place when it is > complete. > > ZFS does not check if data content has been changed while it is being > written > > so a write of the same data will result in a fresh allocation based on > its > > Copy On Write ("COW") design. Writing a whole new file obviously > > significantly increases the number of blocks which are written. > Requesting > > that rsync only write to the file for the blocks which have changed > reduces > > the total number of blocks which get written. > > > > The above helps quite a lot when using snapshots since then fewer blocks > are > > in the snapshots. > > > > I have never tried --delete-during so I can't comment on that. > > > > Bob > > -- > > Bob Friesenhahn > > bfriesen@simple.dallas.tx.us, > http://www.simplesystems.org/users/bfriesen/ > > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ > > Public Key, > http://www.simplesystems.org/users/bfriesen/public-key.txt > > > > > > > ---------------------------- > Andy Thomas, > Time Domain Systems > > Tel: +44 (0)7866 556626 > http://www.time-domain.co.uk > > --00000000000011651005eda74014 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Now that you have it working with rsync, you should look = into using ZFS send/recv as an alternative. You should find it finishes a l= ot quicker than rsync, although it does require a bit more scripting know-h= ow (especially if you want to use restartable/interruptible transfers, or u= se a transport other than SSH for better throughout).

=
ZFS send/recv works "below" the filesyste= m later today rsync works at. ZFS knows which individual blocks on disk hav= e changed between snapshots and only transfers those blocks. There's no= file comparisons and hash computations to work out between the hosts.

Transferring the initial sna= pshot takes a long time, though, as it has to transfer the entire filesyste= m across. Transferring individual snapshots after that takes very little ti= me. It's similar to doing a "full" backup, and then "inc= rementals".

When tr= ansferring data between ZFS pools with similar filesystem hierarchies, you = really should consider send/recv.

Cheers,
Freddie

Typos due to= smartphone keyboard.

On Thu., Nov. 17, 2022, 12:50 a.m. a= ndy thomas, <andy@time-domain.= co.uk> wrote:
I thought I wo= uld report back that changed my rsync options from '-Wav
--delete' to '-av --inplace --no-whole-file --delete' has made = a
significant difference, with mirrored directory sizes on the slave server <= br> now falling and approaching the original sizes on the master. The only
downside is that since whole-file replication is obviously a lot faster than updating the changed parts of individual files, mirroring is now
taking longer than 24 hours so this will be changed to every few days or even weekly when more is known about user behaviour on the master server.
Andy

On Sun, 13 Nov 2022, Bob Friesenhahn wrote:

> On Sun, 13 Nov 2022, Mark Saad wrote:
>>>
>> Bob are you saying when the target is zfs --inplace --no-whole-fil= e helps
>> or just in general when you have
>> large files ?=C2=A0 Also have you tried using --delete-during / --= delete-after
>> ?
>
> The '-inplace --no-whole-file' updates the file blocks if they= have changed
> (comparing the orgin blocks with the existing mirror blocks) rather th= an
> creating a new copy of the file and moving it into place when it is co= mplete.
> ZFS does not check if data content has been changed while it is being = written
> so a write of the same data will result in a fresh allocation based on= its
> Copy On Write ("COW") design.=C2=A0 Writing a whole new file= obviously
> significantly increases the number of blocks which are written.=C2=A0 = Requesting
> that rsync only write to the file for the blocks which have changed re= duces
> the total number of blocks which get written.
>
> The above helps quite a lot when using snapshots since then fewer bloc= ks are
> in the snapshots.
>
> I have never tried --delete-during so I can't comment on that.
>
> Bob
> --
> Bob Friesenhahn
> bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
> GraphicsMagick Maintainer,=C2=A0 =C2=A0 http://www.Gra= phicsMagick.org/
> Public Key,=C2=A0 =C2=A0 =C2=A0http://www.simplesystems.org/users/bfriesen/public-key.txt
>
>


----------------------------
Andy Thomas,
Time Domain Systems

Tel: +44 (0)7866 556626
http://www.time-domain.co.uk

--00000000000011651005eda74014--