ZFS volume does not seem to free space for deleted files

[accounts-ag]

Hi,

I'm trying to clean up a zfs volume used by a FreeBSD jail:

NAME            USED  AVAIL  REFER  MOUNTPOINT
tank/jail/www   116G  68.3G  4.57G  /tank/jail/www

The USED space was probably accurate before I freed a lot of space on
this device by cleaning up /var/tmp and some (a lot...) system mails
which accumulated over years. According to "du" this at least freed
+50GB. However, somehow ZFS does not free that yet.

In the stopped jail starting from /tank/jail/www, "du" (or ncdu to be
precise) reports:

Total disk usage:   4.6 GiB  Apparent size:   4.6 GiB  Items: 76516

This seems to correspond with what is listed as REFER.

So I checked space usage over everything on tank/jail/www:

# zfs list -t all -o space -r tank/jail/www
NAME                                           AVAIL   USED USEDSNAP  USEDDS
tank/jail/www                                  68.3G   116G     112G   4.57G
tank/jail/www at monthly-2015-07-01_00.00.00--3m      -  29.0M         -
    -
tank/jail/www at daily-2015-07-01_00.00.00--2w        -  27.8M         -
    -
tank/jail/www at daily-2015-07-02_00.00.00--2w        -  1.22G         -
    -
tank/jail/www at daily-2015-07-03_00.00.00--2w        -  1.27G         -
    -
tank/jail/www at daily-2015-07-04_00.00.00--2w        -  1.52G         -
    -
tank/jail/www at daily-2015-07-05_00.00.00--2w        -  62.6M         -
    -
tank/jail/www at weekly-2015-07-05_00.00.00--1m       -  62.5M         -
    -
tank/jail/www at daily-2015-07-06_00.00.00--2w        -  1.51G         -
    -
tank/jail/www at daily-2015-07-07_00.00.00--2w        -  1.38G         -
    -
tank/jail/www at daily-2015-07-08_00.00.00--2w        -  1.39G         -
    -
tank/jail/www at daily-2015-07-09_00.00.00--2w        -  1.49G         -
    -
tank/jail/www at daily-2015-07-10_00.00.00--2w        -  1.17G         -
    -
tank/jail/www at daily-2015-07-11_00.00.00--2w        -  1.33G         -
    -
tank/jail/www at weekly-2015-07-12_00.00.00--1m       -   119M         -
    -
tank/jail/www at daily-2015-07-12_00.00.00--2w        -   117M         -
    -
tank/jail/www at hourly-2015-07-12_04.04.00--2d       -   255M         -
    -
tank/jail/www at hourly-2015-07-12_10.04.00--2d       -   265M         -
    -
tank/jail/www at hourly-2015-07-12_16.04.00--2d       -   267M         -
    -
tank/jail/www at hourly-2015-07-12_22.04.00--2d       -   204M         -
    -
tank/jail/www at daily-2015-07-13_00.00.00--2w        -   257M         -
    -
tank/jail/www at hourly-2015-07-13_04.04.00--2d       -   257M         -
    -
tank/jail/www at hourly-2015-07-13_10.04.00--2d       -   260M         -
    -
tank/jail/www at hourly-2015-07-13_16.04.00--2d       -  1.70G         -
    -
tank/jail/www at hourly-2015-07-13_22.04.00--2d       -  48.4M         -
    -
tank/jail/www at daily-2015-07-14_00.00.00--2w        -  32.4M         -
    -
tank/jail/www at hourly-2015-07-14_04.04.00--2d       -  20.0M         -
    -
tank/jail/www at hourly-2015-07-14_10.04.00--2d       -  18.9M         -
    -

So according to my calculation it is using less than 20GB of space for
snapshots. Which does not make much sense to me as I clearly do expect
the deleted files to show up somewhere (I did it on 2015-07-13). So how
can the big difference to USED be explained which reports 116G in total?

Would I have to delete all snapshots to properly free the disk space? I
was unmounting the volume meanwhile and remounting it but that didn't
change anything.

zpool info:

# zpool list -o name,size,allocated,free,freeing
NAME   SIZE  ALLOC   FREE  FREEING
tank   295G   222G  72.9G        0

thanks

regards

Adrian