zfs: the exponential file system from hell

[Ronald Klop]

On Fri, 27 Sep 2013 14:29:38 +0200, Attila Nagy <bra at fsn.hu> wrote:

> Hi,
>
> Did anyone try to fill a zpool with multiple zfs in it and graph the  
> space accounted by df and zpool list?
> If not, here it is:
> https://picasaweb.google.com/104147045962330059540/FreeBSDZfsVsDf#5928271443977601554
>
> The zpool in question:
>          NAME        STATE     READ WRITE CKSUM
>          mnt         ONLINE       0     0     0
>            raidz2-0  ONLINE       0     0     0
>              da1     ONLINE       0     0     0
>              da2     ONLINE       0     0     0
>              da3     ONLINE       0     0     0
>              da4     ONLINE       0     0     0
>            raidz2-1  ONLINE       0     0     0
>              da5     ONLINE       0     0     0
>              da6     ONLINE       0     0     0
>              da7     ONLINE       0     0     0
>              da8     ONLINE       0     0     0
>            raidz2-2  ONLINE       0     0     0
>              da9     ONLINE       0     0     0
>              da10    ONLINE       0     0     0
>              da11    ONLINE       0     0     0
>              da13    ONLINE       0     0     0
>
> And the zfs-es on it:
> NAME     USED  AVAIL  REFER  MOUNTPOINT
> mnt     23.3G      0  53.8K  /mnt
> mnt/p1  3.89G      0  3.89G  /mnt/p1
> mnt/p2  3.89G      0  3.89G  /mnt/p2
> mnt/p3  3.89G      0  3.89G  /mnt/p3
> mnt/p4  3.89G      0  3.89G  /mnt/p4
> mnt/p5  3.89G      0  3.89G  /mnt/p5
> mnt/p6  3.89G      0  3.89G  /mnt/p6
>
> I have evenly filled the six zfs in 1/100th percents of the full space  
> available, and graphed the results of zpool list's capacity and df's  
> capacity.
> The x scale is the real space usage in percents.
>
> It's quite annoying when df says the file systems are 20% full, while in  
> reality, they are at 60%.
>
> Any chance that it will be solved?

So, if I understand you correctly, you compare the output of free space of  
zpool list (of the whole pool) to the output of df of one of size zfs's.
So the conclusion is that zpool list shows you how much space there is in  
the pool and df shows the  
amount-of-space-of-one-zfs-if-you-don't-do-change-other-zfs's-in-the-pool-in-the-meantime.

This is not going to be fixed. It is by design. Choose the best tool to  
measure what you want to know.
But I understand it is a bit confusing if you come from traditional  
fixed-size filesystems.
You can emulate a fixed-size filesystem by using quota's. Give all zfs's  
1/6 of the space in quota and your df will work perfectly.

NB: I'm not a ZFS developer so I don't have an authoritative opinion about  
all this.

Regards,
Ronald.