From nobody Fri Nov 11 20:20:37 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4N897Y5lBRz4fvSS for ; Fri, 11 Nov 2022 20:20:41 +0000 (UTC) (envelope-from andy@time-domain.co.uk) Received: from mail0.time-domain.net (mail0.time-domain.net [62.3.122.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4N897X2s9Gz4NQf for ; Fri, 11 Nov 2022 20:20:40 +0000 (UTC) (envelope-from andy@time-domain.co.uk) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of andy@time-domain.co.uk designates 62.3.122.138 as permitted sender) smtp.mailfrom=andy@time-domain.co.uk; dmarc=none Received: from mail0.time-domain.net (localhost [127.0.0.1]) by mail0.time-domain.net (8.15.2/8.15.2) with ESMTP id 2ABKKbIC030680; Fri, 11 Nov 2022 20:20:37 GMT (envelope-from andy@time-domain.co.uk) Received: from localhost (andy-tds@localhost) by mail0.time-domain.net (8.15.2/8.15.2/Submit) with ESMTP id 2ABKKbt1030677; Fri, 11 Nov 2022 20:20:37 GMT (envelope-from andy@time-domain.co.uk) X-Authentication-Warning: mail0.time-domain.net: andy-tds owned process doing -bs Date: Fri, 11 Nov 2022 20:20:37 +0000 (GMT) From: andy thomas X-X-Sender: andy-tds@mail0.time-domain.net To: Mehmet Erol Sanliturk cc: andy thomas , freebsd-fs@freebsd.org Subject: Re: Odd behaviour of two identical ZFS servers mirroring via rsync In-Reply-To: Message-ID: References: User-Agent: Alpine 2.22 (BSF 395 2020-01-19) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="951801389-1876059679-1668198037=:30520" X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.30 / 15.00]; CTYPE_MIXED_BOGUS(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; R_SPF_ALLOW(-0.20)[+ip4:62.3.122.136/29:c]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; FREEMAIL_TO(0.00)[gmail.com]; ARC_NA(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; ASN(0.00)[asn:13037, ipnet:62.3.64.0/18, country:GB]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[andy]; TO_DN_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; HAS_XAW(0.00)[]; TAGGED_RCPT(0.00)[]; DMARC_NA(0.00)[time-domain.co.uk]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4N897X2s9Gz4NQf X-ThisMailContainsUnwantedMimeParts: N This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --951801389-1876059679-1668198037=:30520 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8BIT Yes, I can confirm the rsync --delete option is being used and in fact, 'du' reports some of the mirrored folders as having identical sizes on both servers, mainly those containing only small amounts of data. It seems almost as if ZFS is not freeing up blocks when rsync has deleted or shrank files, leaving unwanted blocks lurking around in the folder that 'du' then discovers and adds to its tally when it works out the space usage of that folder! I suppose I could always destroy the zfs dataset on the mirror server & resync the whole thing but will take days to complete even over a 10 Gbit/s network link (the servers ought to be upgraded to FBSD 13.1 as well). Andy On Fri, 11 Nov 2022, Mehmet Erol Sanliturk wrote: > > , Nov 11, 2022 at 8:42 PM andy thomas wrote: > I have two identical servers, called clustor2 and > clustor-backup, each > with a ZFS RAIDZ-1 pool containing 9 SAS hard disks plus one > spare and two > SSDs for the ZIL and ARC functions. clustor2 stores user data > from a > HPC while clustor2-backup uses rsync to mirrors all the data > from clustor2 > every 24 hours. > > However, the disk usage on the mirror server is considerably > more than on > the other server - attached is a screenshot showing the two > servers side > by side, with the mirror server on the right, and displaying the > contents > of the same subdirectory choen at random (named 'ratio_10.0' in > this > instance); as you can see, the sizes of the files within each of > the > folders are identical but 'du' reports very different > space usages for each folder and 'zpool list' also reports a > significant > difference in ZFS pool size. > > I'm not sure if this is relevant but both servers have ZFS pools > with no > compression although lz4 compression is enabled on the ZFS > filesystems & > both run FreeBSD 11.3 with ZFS version 5. > > Perhaps using zfs send/receive instead of rsync for mirroring > might solve > this disparity? > > Thanks in advance for any suggestions, > > Andy > > > > > Your question I am understanding the following points . > > > > I am using  rsync  in Fedora Linux . > > There are  parameters of  rsync  such as > >  --delete > > to delete files from the destination drive when they do not exist in the > source drive . > > > Please carefully scan  rsync  parameters and use suitable ones for your > application . > > > If  a parameter like  --delete  is not used , rsync  copies new files from > the source drive and > it does not delete any files from the destination drive . > > > With my best wishes for all . > > > Mehmet Erol Sanliturk > > > > > > >   > > ---------------------------- Andy Thomas, Time Domain Systems Tel: +44 (0)7866 556626 http://www.time-domain.co.uk --951801389-1876059679-1668198037=:30520--