From nobody Sun Nov 13 17:46:42 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4N9Kd23vvXz4hlvZ for ; Sun, 13 Nov 2022 17:46:46 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from smtp.simplesystems.org (smtp.simplesystems.org [65.66.246.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4N9Kd16dRhz3w0J for ; Sun, 13 Nov 2022 17:46:45 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Authentication-Results: mx1.freebsd.org; none Received: from scrappy.simplesystems.org (scrappy.simplesystems.org [65.66.246.73]) by smtp.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id 2ADHkgq5026015; Sun, 13 Nov 2022 11:46:42 -0600 (CST) Date: Sun, 13 Nov 2022 11:46:42 -0600 (CST) From: Bob Friesenhahn X-X-Sender: bfriesen@scrappy.simplesystems.org To: Mark Saad cc: freebsd-fs@FreeBSD.org Subject: Re: Odd behaviour of two identical ZFS servers mirroring via rsync In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (GSO 67 2015-01-07) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (smtp.simplesystems.org [65.66.246.90]); Sun, 13 Nov 2022 11:46:42 -0600 (CST) X-Rspamd-Queue-Id: 4N9Kd16dRhz3w0J X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:7018, ipnet:65.64.0.0/13, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Sun, 13 Nov 2022, Mark Saad wrote: >> > Bob are you saying when the target is zfs --inplace --no-whole-file helps > or just in general when you have > large files ? Also have you tried using --delete-during / --delete-after ? The '-inplace --no-whole-file' updates the file blocks if they have changed (comparing the orgin blocks with the existing mirror blocks) rather than creating a new copy of the file and moving it into place when it is complete. ZFS does not check if data content has been changed while it is being written so a write of the same data will result in a fresh allocation based on its Copy On Write ("COW") design. Writing a whole new file obviously significantly increases the number of blocks which are written. Requesting that rsync only write to the file for the blocks which have changed reduces the total number of blocks which get written. The above helps quite a lot when using snapshots since then fewer blocks are in the snapshots. I have never tried --delete-during so I can't comment on that. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt