[Bug 290205] ZFS: file stuck in delete queue
Date: Mon, 13 Oct 2025 13:40:49 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=290205
Bug ID: 290205
Summary: ZFS: file stuck in delete queue
Product: Base System
Version: 14.3-RELEASE
Hardware: Any
OS: Any
Status: New
Severity: Affects Only Me
Priority: ---
Component: kern
Assignee: bugs@FreeBSD.org
Reporter: ml@netfence.it
I've got a problem where a file is seemingly in the delete queue and has been
there for months.
This isn't an issue with space (I have plenty of), but "zfs send"ing the
incriminated pool produces an overly inflated file (51GB after compression,
when it should be around 2GB).
The box is FreeBSD 14.3/amd64, fully patched.
There are two pools, all of then in good HEALTH and with "No known data
errors":
# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH
ALTROOT
vm 446G 329G 117G - - 49% 73% 1.00x ONLINE -
zroot 3.56T 1.52T 2.04T - - 19% 42% 1.00x ONLINE -
#zfs list -o
name,mountpoint,canmount,refer,used,usedbychildren,usedbydataset,usedbyrefreservation,usedbysnapshots
zroot /zroot on
88K 1.52T 1.52T 88K 0B 536K
zroot/ROOT none on
88K 65.4G 65.4G 88K 0B 0B
zroot/ROOT/default / noauto
62.1G 65.4G 0B 62.1G 0B 3.23G
zroot/ezjail /usr/jails on
136K 1.41T 1.41T 136K 0B 328K
...
As you can see zroot/ROOT/default has 62.1G used by dataset, yet only 3.3G seem
to exist when using standard file system tools:
# du -d 0 -x -h /
3.3G /
With the help of the mailing list, we found out the following:
# zdb -ddd zroot/ROOT/default 0:-1:A |sort -h -b -k 5 |tail -5
163861 3 128K 128K 77.1M 512 178M 100.00 ZFS plain file
130966 3 128K 128K 96.4M 512 262M 100.00 ZFS plain file
163976 3 128K 128K 139M 512 335M 100.00 ZFS plain file
130956 3 128K 128K 145M 512 343M 100.00 ZFS plain file
360 3 128K 128K 58.8G 512 90.0G 100.00 ZFS plain file
# zdb -ddddd zroot/ROOT/default 360
Dataset zroot/ROOT/default [ZPL], ID 89, cr_txg 8, 62.1G, 107125 objects,
rootbp DVA[0]=<0:103e1032000:1000> DVA[1]=<0:4365c65000:1000> [L0 DMU objset]
fletcher4 uncompressed unencrypted LE contiguous unique double size=800L/800P
birth=3
7011214L/37011214P fill=107125
cksum=0000000e1765a187:00000fbba696e8af:000aec7540e5bc65:05a9323d03d6cb70
Object lvl iblk dblk dsize dnsize lsize %full type
360 3 128K 128K 58.8G 512 90.0G 100.00 ZFS plain file
168 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 737498
path on delete queue
uid 0
gid 0
...
So this is a 58.8GB file that has, from the user perspective, been deleted, but
it's still referenced by ZFS.
Rebooting (several times) did not help.
Deleting all zroot/ROOT/default's snapshot was useless.
In case it is needed:
# zpool get all zroot
NAME PROPERTY VALUE SOURCE
zroot size 3.56T -
zroot capacity 42% -
zroot altroot - default
zroot health ONLINE -
zroot guid 10368181361075995187 -
zroot version - default
zroot bootfs zroot/ROOT/default local
zroot delegation on default
zroot autoreplace off default
zroot cachefile - default
zroot failmode wait default
zroot listsnapshots off default
zroot autoexpand off default
zroot dedupratio 1.00x -
zroot free 2.04T -
zroot allocated 1.52T -
zroot readonly off -
zroot ashift 0 default
zroot comment - default
zroot expandsize - -
zroot freeing 0 -
zroot fragmentation 19% -
zroot leaked 0 -
zroot multihost off default
zroot checkpoint - -
zroot load_guid 8455461369367935358 -
zroot autotrim off default
zroot compatibility off default
zroot bcloneused 0 -
zroot bclonesaved 0 -
zroot bcloneratio 1.00x -
zroot feature@async_destroy enabled local
zroot feature@empty_bpobj active local
zroot feature@lz4_compress active local
zroot feature@multi_vdev_crash_dump enabled local
zroot feature@spacemap_histogram active local
zroot feature@enabled_txg active local
zroot feature@hole_birth active local
zroot feature@extensible_dataset active local
zroot feature@embedded_data active local
zroot feature@bookmarks enabled local
zroot feature@filesystem_limits enabled local
zroot feature@large_blocks enabled local
zroot feature@large_dnode enabled local
zroot feature@sha512 enabled local
zroot feature@skein enabled local
zroot feature@edonr disabled local
zroot feature@userobj_accounting disabled local
zroot feature@encryption disabled local
zroot feature@project_quota disabled local
zroot feature@device_removal enabled local
zroot feature@obsolete_counts enabled local
zroot feature@zpool_checkpoint enabled local
zroot feature@spacemap_v2 active local
zroot feature@allocation_classes disabled local
zroot feature@resilver_defer disabled local
zroot feature@bookmark_v2 disabled local
zroot feature@redaction_bookmarks disabled local
zroot feature@redacted_datasets disabled local
zroot feature@bookmark_written disabled local
zroot feature@log_spacemap disabled local
zroot feature@livelist disabled local
zroot feature@device_rebuild disabled local
zroot feature@zstd_compress disabled local
zroot feature@draid disabled local
zroot feature@zilsaxattr disabled local
zroot feature@head_errlog disabled local
zroot feature@blake3 disabled local
zroot feature@block_cloning disabled local
zroot feature@vdev_zaps_v2 disabled local
# zfs get all zroot/ROOT/default
NAME PROPERTY VALUE SOURCE
zroot/ROOT/default type filesystem -
zroot/ROOT/default creation Sun Dec 15 16:58 2019 -
zroot/ROOT/default used 65.4G -
zroot/ROOT/default available 1.93T -
zroot/ROOT/default referenced 62.1G -
zroot/ROOT/default compressratio 1.58x -
zroot/ROOT/default mounted yes -
zroot/ROOT/default quota none default
zroot/ROOT/default reservation none default
zroot/ROOT/default recordsize 128K default
zroot/ROOT/default mountpoint / local
zroot/ROOT/default sharenfs off default
zroot/ROOT/default checksum on default
zroot/ROOT/default compression lz4 inherited
from zroot
zroot/ROOT/default atime off inherited
from zroot
zroot/ROOT/default devices on default
zroot/ROOT/default exec on default
zroot/ROOT/default setuid on default
zroot/ROOT/default readonly off local
zroot/ROOT/default jailed off default
zroot/ROOT/default snapdir hidden default
zroot/ROOT/default aclmode discard default
zroot/ROOT/default aclinherit restricted default
zroot/ROOT/default createtxg 8 -
zroot/ROOT/default canmount noauto local
zroot/ROOT/default xattr on default
zroot/ROOT/default copies 1 default
zroot/ROOT/default version 5 -
zroot/ROOT/default utf8only off -
zroot/ROOT/default normalization none -
zroot/ROOT/default casesensitivity sensitive -
zroot/ROOT/default vscan off default
zroot/ROOT/default nbmand off default
zroot/ROOT/default sharesmb off default
zroot/ROOT/default refquota none default
zroot/ROOT/default refreservation none default
zroot/ROOT/default guid 13999796839276270198 -
zroot/ROOT/default primarycache all default
zroot/ROOT/default secondarycache all default
zroot/ROOT/default usedbysnapshots 3.23G -
zroot/ROOT/default usedbydataset 62.1G -
zroot/ROOT/default usedbychildren 0B -
zroot/ROOT/default usedbyrefreservation 0B -
zroot/ROOT/default logbias latency default
zroot/ROOT/default objsetid 89 -
zroot/ROOT/default dedup off default
zroot/ROOT/default mlslabel none default
zroot/ROOT/default sync standard default
zroot/ROOT/default dnodesize legacy default
zroot/ROOT/default refcompressratio 1.56x -
zroot/ROOT/default written 1.16M -
zroot/ROOT/default logicalused 103G -
zroot/ROOT/default logicalreferenced 96.7G -
zroot/ROOT/default volmode default default
zroot/ROOT/default filesystem_limit none default
zroot/ROOT/default snapshot_limit none default
zroot/ROOT/default filesystem_count none default
zroot/ROOT/default snapshot_count none default
zroot/ROOT/default snapdev hidden default
zroot/ROOT/default acltype nfsv4 default
zroot/ROOT/default context none default
zroot/ROOT/default fscontext none default
zroot/ROOT/default defcontext none default
zroot/ROOT/default rootcontext none default
zroot/ROOT/default relatime on default
zroot/ROOT/default redundant_metadata all default
zroot/ROOT/default overlay on default
zroot/ROOT/default encryption off default
zroot/ROOT/default keylocation none default
zroot/ROOT/default keyformat none default
zroot/ROOT/default pbkdf2iters 0 default
zroot/ROOT/default special_small_blocks 0 default
zroot/ROOT/default snapshots_changed Sun Oct 5 17:00:01 2025 -
zroot/ROOT/default prefetch all default
zroot/ROOT/default autobackup:auto_zroot true inherited
from zroot
--
You are receiving this mail because:
You are the assignee for the bug.