From nobody Thu Nov 17 08:50:20 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NCYXQ3GX9z4hYK1 for ; Thu, 17 Nov 2022 08:50:30 +0000 (UTC) (envelope-from andy@time-domain.co.uk) Received: from mail0.time-domain.net (mail0.time-domain.net [62.3.122.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4NCYXP22rQz45SB for ; Thu, 17 Nov 2022 08:50:28 +0000 (UTC) (envelope-from andy@time-domain.co.uk) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of andy@time-domain.co.uk designates 62.3.122.138 as permitted sender) smtp.mailfrom=andy@time-domain.co.uk; dmarc=none Received: from mail0.time-domain.net (localhost [127.0.0.1]) by mail0.time-domain.net (8.15.2/8.15.2) with ESMTP id 2AH8oLC7046302; Thu, 17 Nov 2022 08:50:21 GMT (envelope-from andy@time-domain.co.uk) Received: from localhost (andy-tds@localhost) by mail0.time-domain.net (8.15.2/8.15.2/Submit) with ESMTP id 2AH8oKlG046299; Thu, 17 Nov 2022 08:50:21 GMT (envelope-from andy@time-domain.co.uk) X-Authentication-Warning: mail0.time-domain.net: andy-tds owned process doing -bs Date: Thu, 17 Nov 2022 08:50:20 +0000 (GMT) From: andy thomas X-X-Sender: andy-tds@mail0.time-domain.net To: Bob Friesenhahn cc: Mark Saad , freebsd-fs@FreeBSD.org Subject: Re: Odd behaviour of two identical ZFS servers mirroring via rsync In-Reply-To: Message-ID: References: User-Agent: Alpine 2.22 (BSF 395 2020-01-19) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Spamd-Result: default: False [-1.96 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_SPAM_SHORT(0.34)[0.337]; R_SPF_ALLOW(-0.20)[+ip4:62.3.122.136/29]; MIME_GOOD(-0.10)[text/plain]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@FreeBSD.org]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DMARC_NA(0.00)[time-domain.co.uk]; HAS_XAW(0.00)[]; FREEFALL_USER(0.00)[andy]; ASN(0.00)[asn:13037, ipnet:62.3.64.0/18, country:GB]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4NCYXP22rQz45SB X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N I thought I would report back that changed my rsync options from '-Wav --delete' to '-av --inplace --no-whole-file --delete' has made a significant difference, with mirrored directory sizes on the slave server now falling and approaching the original sizes on the master. The only downside is that since whole-file replication is obviously a lot faster than updating the changed parts of individual files, mirroring is now taking longer than 24 hours so this will be changed to every few days or even weekly when more is known about user behaviour on the master server. Andy On Sun, 13 Nov 2022, Bob Friesenhahn wrote: > On Sun, 13 Nov 2022, Mark Saad wrote: >>> >> Bob are you saying when the target is zfs --inplace --no-whole-file helps >> or just in general when you have >> large files ? Also have you tried using --delete-during / --delete-after >> ? > > The '-inplace --no-whole-file' updates the file blocks if they have changed > (comparing the orgin blocks with the existing mirror blocks) rather than > creating a new copy of the file and moving it into place when it is complete. > ZFS does not check if data content has been changed while it is being written > so a write of the same data will result in a fresh allocation based on its > Copy On Write ("COW") design. Writing a whole new file obviously > significantly increases the number of blocks which are written. Requesting > that rsync only write to the file for the blocks which have changed reduces > the total number of blocks which get written. > > The above helps quite a lot when using snapshots since then fewer blocks are > in the snapshots. > > I have never tried --delete-during so I can't comment on that. > > Bob > -- > Bob Friesenhahn > bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ > Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt > > ---------------------------- Andy Thomas, Time Domain Systems Tel: +44 (0)7866 556626 http://www.time-domain.co.uk