Re: ZFS missing space

From: Frank Leonhardt <freebsd-doc_at_fjl.co.uk>
Date: Wed, 12 Feb 2025 21:48:45 UTC
On 12/02/2025 14:14, mike tancsa wrote:
> On 2/12/2025 5:55 AM, Frank Leonhardt wrote:
>> I've noticed space "go missing" on ZFS before, but not conclusively. 
>> This time it's happened on a brand new setup I'm doing some testing on.
>> As this is a "clean" system I can't figure out where the discrepancy 
>> could possibly be coming from. This is beyond the slop value.
>>
>> Any ideas anyone?
>>
> What do
>
> zfs list -t snapshot -sused -r zr
> zpool get all
>
> show ?
>
> does
> fstat
> show any open files at the time ?
>
>     ---Mike

There are no snapshots. zfs list -t all would have shown them and 
anything else.

I see where you're going with fstat, but it's been restarted (several 
times) and there's nothing hanging. But when it rebooted itself it was 
creating a dataset of about the same size as the black hole.

And yes, I did try a scrub :-)

Since then I've tested all the drives by reading every block and I'm 
running memory soak tests. It's ECC RAM anyway.

-----

root@zfs2:/ #  zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP DEDUP    
HEALTH  ALTROOT
zr    7.27T  10.1G  7.26T        -         -     0%     0% 1.00x    
ONLINE  -

------

root@zfs2:/ # zpool status
   pool: zr
  state: ONLINE
   scan: scrub repaired 0B in 00:00:36 with 0 errors on Wed Feb 12 
10:36:17 2025
config:

         NAME        STATE     READ WRITE CKSUM
         zr          ONLINE       0     0     0
           raidz1-0  ONLINE       0     0     0
             da0p3   ONLINE       0     0     0
             da1p3   ONLINE       0     0     0
             da2p3   ONLINE       0     0     0
             da3p3   ONLINE       0     0     0

errors: No known data errors

-------

root@zfs2:/ # zfs list
NAME              USED  AVAIL  REFER  MOUNTPOINT
zr               7.33G  5.15T   140K  /zr
zr/ROOT          5.12G  5.15T   140K  none
zr/ROOT/default  5.12G  5.15T  5.12G  /
zr/data           140K  5.15T   140K  /data
zr/home           343K  5.15T   140K  /home
zr/home/fjl       203K  5.15T   203K  /home/fjl
zr/tmp            140K  5.15T   140K  /tmp
zr/usr           2.19G  5.15T   140K  /usr
zr/usr/ports     1.16G  5.15T  1.16G  /usr/ports
zr/usr/src       1.03G  5.15T  1.03G  /usr/src
zr/var           1.12M  5.15T   140K  /var
zr/var/audit      140K  5.15T   140K  /var/audit
zr/var/crash      140K  5.15T   140K  /var/crash
zr/var/log        407K  5.15T   407K  /var/log
zr/var/mail       180K  5.15T   180K  /var/mail
zr/var/tmp        140K  5.15T   140K  /var/tmp

-----------

zfs list and zpool list agree on what's used, just not what's free.

It's like there's a hidden broken dataset that's allocated space but 
isn't listed.

Thanks, Frank.