From nobody Sat Nov 12 08:24:27 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4N8TBl24NVz4g55y for ; Sat, 12 Nov 2022 08:24:31 +0000 (UTC) (envelope-from andy@time-domain.co.uk) Received: from mail0.time-domain.net (mail0.time-domain.net [62.3.122.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4N8TBk5npDz3GVn for ; Sat, 12 Nov 2022 08:24:30 +0000 (UTC) (envelope-from andy@time-domain.co.uk) Authentication-Results: mx1.freebsd.org; none Received: from mail0.time-domain.net (localhost [127.0.0.1]) by mail0.time-domain.net (8.15.2/8.15.2) with ESMTP id 2AC8OSGM039820; Sat, 12 Nov 2022 08:24:28 GMT (envelope-from andy@time-domain.co.uk) Received: from localhost (andy-tds@localhost) by mail0.time-domain.net (8.15.2/8.15.2/Submit) with ESMTP id 2AC8OSRb039817; Sat, 12 Nov 2022 08:24:28 GMT (envelope-from andy@time-domain.co.uk) X-Authentication-Warning: mail0.time-domain.net: andy-tds owned process doing -bs Date: Sat, 12 Nov 2022 08:24:27 +0000 (GMT) From: andy thomas X-X-Sender: andy-tds@mail0.time-domain.net To: Bob Friesenhahn cc: andy thomas , freebsd-fs@FreeBSD.org Subject: Re: Odd behaviour of two identical ZFS servers mirroring via rsync In-Reply-To: Message-ID: References: User-Agent: Alpine 2.22 (BSF 395 2020-01-19) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 4N8TBk5npDz3GVn X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:13037, ipnet:62.3.64.0/18, country:GB] X-ThisMailContainsUnwantedMimeParts: N Thank you for the suggestions, I'll set up a pair of test servers and experiment with adjusting the rsync block size to match the 128k ZFS record size, noting disk usage on both for varying file sizes. Buffering could well be an issue here with data on the server being mirrrored contantly changing within the HPC it supports (30 Linux compute nodes with up to 700 simultaneous user jobs) and this might be something I will just have to live with. Andy On Fri, 11 Nov 2022, Bob Friesenhahn wrote: > On Fri, 11 Nov 2022, andy thomas wrote: >> >> It seems almost as if ZFS is not freeing up blocks when rsync has deleted >> or shrank files, leaving unwanted blocks lurking around in the folder that >> 'du' then discovers and adds to its tally when it works out the space usage >> of that folder! > > This would be completely expected behavior if zfs snapshots are used. > > The rsync block sizes can be adjusted to be a better match for zfs block > sizes (e.g. 128k). For example, zfs will do a 'sync' to write new data to > disk and it will help if all of the data in an new/updated zfs block is > provided at the time of the 'sync' (rather than 1/4 or 1/2 of the block). > > Network buffering can also be a factor since it effects the timing of data > delivery to the backup server. If the sending side tends to stall, then add > more buffering. > > Bob > -- > Bob Friesenhahn > bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ > Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt > > ---------------------------- Andy Thomas, Time Domain Systems Tel: +44 (0)7866 556626 http://www.time-domain.co.uk