Re: Cannot find out what uses space in ZFS dataset
Date: Thu, 18 Sep 2025 21:22:24 UTC
On 9/18/25 03:10, Andrea Venturoli wrote: > Hello. > > I've got a urgent trouble I'm trying to solve. > I need to take backups of a server: they should be around 4-5 GBs, but > the end up close to 100GB!!! > > The trouble is with the zroot/ROOT/default dataset: >> # cd / >> # du -d 0 -h -x >> 3.3G . > > Yet: >> root@chet:/ # zfs list|grep ROOT >> zroot/ROOT 70.9G 1.94T 88K none >> zroot/ROOT/default 70.9G 1.94T 62.1G / > > So ho do we go from 3.3G to 62.1G??? > There are snapshots, but I'm using plain "zfs send", so they should not > matter... > > > >> # zfs get all zroot/ROOT/default >> NAME PROPERTY VALUE >> SOURCE >> zroot/ROOT/default type filesystem - >> zroot/ROOT/default creation Sun Dec 15 16:58 2019 - >> zroot/ROOT/default used 70.9G - >> zroot/ROOT/default available 1.94T - >> zroot/ROOT/default referenced 62.1G - >> zroot/ROOT/default compressratio 1.62x - >> zroot/ROOT/default mounted yes - >> zroot/ROOT/default quota none >> default >> zroot/ROOT/default reservation none >> default >> zroot/ROOT/default recordsize 128K >> default >> zroot/ROOT/default mountpoint / >> local >> zroot/ROOT/default sharenfs off >> default >> zroot/ROOT/default checksum on >> default >> zroot/ROOT/default compression lz4 >> inherited from zroot >> zroot/ROOT/default atime off >> inherited from zroot >> zroot/ROOT/default devices on >> default >> zroot/ROOT/default exec on >> default >> zroot/ROOT/default setuid on >> default >> zroot/ROOT/default readonly off >> local >> zroot/ROOT/default jailed off >> default >> zroot/ROOT/default snapdir hidden >> default >> zroot/ROOT/default aclmode discard >> default >> zroot/ROOT/default aclinherit restricted >> default >> zroot/ROOT/default createtxg 8 - >> zroot/ROOT/default canmount noauto >> local >> zroot/ROOT/default xattr on >> default >> zroot/ROOT/default copies 1 >> default >> zroot/ROOT/default version 5 - >> zroot/ROOT/default utf8only off - >> zroot/ROOT/default normalization none - >> zroot/ROOT/default casesensitivity sensitive - >> zroot/ROOT/default vscan off >> default >> zroot/ROOT/default nbmand off >> default >> zroot/ROOT/default sharesmb off >> default >> zroot/ROOT/default refquota none >> default >> zroot/ROOT/default refreservation none >> default >> zroot/ROOT/default guid 13999796839276270198 - >> zroot/ROOT/default primarycache all >> default >> zroot/ROOT/default secondarycache all >> default >> zroot/ROOT/default usedbysnapshots 8.76G - >> zroot/ROOT/default usedbydataset 62.1G - >> zroot/ROOT/default usedbychildren 0B - >> zroot/ROOT/default usedbyrefreservation 0B - >> zroot/ROOT/default logbias latency >> default >> zroot/ROOT/default objsetid 89 - >> zroot/ROOT/default dedup off >> default >> zroot/ROOT/default mlslabel none >> default >> zroot/ROOT/default sync standard >> default >> zroot/ROOT/default dnodesize legacy >> default >> zroot/ROOT/default refcompressratio 1.56x - >> zroot/ROOT/default written 1.79M - >> zroot/ROOT/default logicalused 114G - >> zroot/ROOT/default logicalreferenced 96.6G - >> zroot/ROOT/default volmode default >> default >> zroot/ROOT/default filesystem_limit none >> default >> zroot/ROOT/default snapshot_limit none >> default >> zroot/ROOT/default filesystem_count none >> default >> zroot/ROOT/default snapshot_count none >> default >> zroot/ROOT/default snapdev hidden >> default >> zroot/ROOT/default acltype nfsv4 >> default >> zroot/ROOT/default context none >> default >> zroot/ROOT/default fscontext none >> default >> zroot/ROOT/default defcontext none >> default >> zroot/ROOT/default rootcontext none >> default >> zroot/ROOT/default relatime on >> default >> zroot/ROOT/default redundant_metadata all >> default >> zroot/ROOT/default overlay on >> default >> zroot/ROOT/default encryption off >> default >> zroot/ROOT/default keylocation none >> default >> zroot/ROOT/default keyformat none >> default >> zroot/ROOT/default pbkdf2iters 0 >> default >> zroot/ROOT/default special_small_blocks 0 >> default >> zroot/ROOT/default snapshots_changed Thu Sep 18 12:00:02 2025 - >> zroot/ROOT/default prefetch all >> default >> zroot/ROOT/default autobackup:auto_zroot true >> inherited from zroot > > > > usedbydataset is 62.1G!!! > Where can this come from if du says 3.3??? > > > > bye & Thanks > av. Understand that ZFS datasets are hierarchical -- e.g. filesystems within filesystems. It looks like you have root-on-ZFS. This wiki page describes the filesystems the FreeBSD installer likely created within zpool: https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot#Create_the_ZFS_file_system_hierarchy Run the following command to see the filesystems on zpool: # zfs list -r -t filesystem zpool Datasets can have snapshots. This is why ZFS reports more space used than du reports. Run the following command to see the snapshots on zpool: # zfs list -r -t snapshot zpool freebsd-update takes snapshots of the root filesystem prior to upgrades. If your system has been upgraded many times, you will have many snapshots. These snapshots will consume space. That is why the "referenced" values reported by zfs(8) are larger than the usage values reported by du(1). To do a full backup, you need to send the entire dataset hierarchy and all of its snapshots someplace -- a file, another pool, etc.. This is known as "replication". If you first destroy snapshots you do not want (e.g. "prune snapshots"), the replication stream will be smaller. Given the 1 plus 16 filesystems listed above, doing replication by hand will be tedious and error prone. I have written a few generations of scripts to automate this chore, first in Bourne shell and later in Perl. Alternatively, it looks like there is at least one packaged solution available (untested): 2025-09-18 13:49:04 toor@f5 ~ # pkg search replicate | grep -i zfs zfs-replicate-2.0.2 ZFS Snapshot Replication Script Michael W. Lucas wrote several texts that are worth owning: https://mwl.io/nonfiction/os#af3e https://mwl.io/nonfiction/os#fmse https://mwl.io/nonfiction/os#fmzfs https://mwl.io/nonfiction/os#fmaz Klara Systems has several articles that are worth reading: https://klarasystems.com/articles/?topics=zfs https://klarasystems.com/articles/introduction-to-zfs-replication/ David