weird problem w/ ZFS not reclaiming freed space

[Mike Andrews]

On Fri, 19 Jun 2009, Mike Andrews wrote:

> Somehow I've managed to get ZFS on one of my machines into a state where it 
> won't reclaim all space after deleting files AND snapshots off of it:
> (this is with 7.2-STABLE amd64, compiled June 10)
>
> # ls -la /weird
> total 4
> drwxr-x---   2 mysql  mysql     2 Jun 19 02:42 .
> drwxr-xr-x  29 root   wheel  1024 Jun 19 02:44 ..
>
> # df /weird
> Filesystem   1K-blocks      Used     Avail Capacity  Mounted on
> scotch/weird 282201472 109151232 173050240    39%    /weird
>
> # zfs list scotch/weird
> NAME           USED  AVAIL  REFER  MOUNTPOINT
> scotch/weird   104G   164G   104G  /weird
>
> # zfs list -t snapshot | grep scotch/weird
>
> # zfs get all scotch/weird
> NAME          PROPERTY              VALUE                  SOURCE
> scotch/weird  type                  filesystem             -
> scotch/weird  creation              Wed Jun 17  1:20 2009  -
> scotch/weird  used                  104G                   -
> scotch/weird  available             159G                   -
> scotch/weird  referenced            104G                   -
> scotch/weird  compressratio         1.00x                  -
> scotch/weird  mounted               yes                    -
> scotch/weird  quota                 none                   default
> scotch/weird  reservation           none                   default
> scotch/weird  recordsize            128K                   default
> scotch/weird  mountpoint            /weird                 local
> scotch/weird  sharenfs              off                    default
> scotch/weird  checksum              on                     default
> scotch/weird  compression           off                    default
> scotch/weird  atime                 off                    local
> scotch/weird  devices               on                     default
> scotch/weird  exec                  off                    local
> scotch/weird  setuid                off                    local
> scotch/weird  readonly              off                    default
> scotch/weird  jailed                off                    default
> scotch/weird  snapdir               hidden                 default
> scotch/weird  aclmode               groupmask              default
> scotch/weird  aclinherit            restricted             default
> scotch/weird  canmount              on                     default
> scotch/weird  shareiscsi            off                    default
> scotch/weird  xattr                 off                    temporary
> scotch/weird  copies                1                      default
> scotch/weird  version               3                      -
> scotch/weird  utf8only              off                    -
> scotch/weird  normalization         none                   -
> scotch/weird  casesensitivity       sensitive              -
> scotch/weird  vscan                 off                    default
> scotch/weird  nbmand                off                    default
> scotch/weird  sharesmb              off                    default
> scotch/weird  refquota              none                   default
> scotch/weird  refreservation        none                   default
> scotch/weird  primarycache          all                    default
> scotch/weird  secondarycache        all                    default
> scotch/weird  usedbysnapshots       0                      -
> scotch/weird  usedbydataset         104G                   -
> scotch/weird  usedbychildren        0                      -
> scotch/weird  usedbyrefreservation  0                      -
>
>
> If I then rsync stuff to it, space seems OK, if I continue to rsync to it
> every few hours, the used space grows, even if no snapshots are being taken
> If I do take snapshots, then change stuff, then delete the snapshots, the
> snapshot space does appear to be reclaimed.  Also if I 'zfs destroy' the
> filesystem, the space is correctly reclaimed, but once I create a new one
> and repeat the process, the problem reappears.
>
> I have not had any luck reproducing this on another machine yet, but 
> admittedly haven't tried super hard yet.
>
> Scrubbing the zpool returns no errors.
>
> I'm guessing zdb is my only hope at debugging this, but as I've never used it 
> before and as it seems to dump core whenever I try running it, can someone 
> suggest what I need to check/look for in it?
>
> I did also have a panic a few days ago that, based on the text, might be 
> related (I do have the vmdump and core.txt)
>
> panic: solaris assert: P2PHASE(start, 1ULL << sm->sm_shift) == 0, file: 
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/space_map.c, 
> line: 146
>
> ...for which I have a vmdump and a core.txt if anyone wants to look at it.


Just to update this and move it to the -fs mailing list, removing the 
"--sparse" flag from rsync solves this problem, so the bug has something 
to do with ZFS's handling of sparse files.