Disappearing available space with ZFS...what am I missing?

[Aaron]

On Wed, Sep 29, 2010 at 03:14, krad <kraduk at gmail.com> wrote:
>
>
> On 29 September 2010 10:12, Morgan Wesström
> <freebsd-questions at pp.dyndns.biz> wrote:
>>
>> On 2010-09-29 07:56, Aaron wrote:
>>>
>>> I've created a ZFS pool with zpool create tank raidz ada0 ada1 ada2
>>> ada3, and then I add some additional mountpoints (I think they're
>>> called) using zfs create tank/storage, etc. In zpool list, I see the
>>> pool with 3.62T available. With df -h, I see 2.4T available for tank,
>>> and tank/storage. When I first created tank, it had the 3.62T
>>> available as I expected. What am I missing? I do have compression set
>>> to gzip-9 on tank which gets inherited like I want, don't know if that
>>> would affect anything.
>>>
>>> --Aaron
>>
>> There's nothing wrong here that I can see, you just have to make a
>> distinction between the zfs pool and the filesystems within the pool and I
>> agree it can be confusing at first.
>>
>> The numbers suggest you are using 4 x 1TB (base 10 TB) drives? That equals
>> 3.7TiB (base 2 TB) which is the unit zpool/zfs uses. This is the total
>> amount of space available to the pool and includes all space on all drives
>> in the pool. Nothing strange so far.
>>
>> Now, since you've told zpool to create filesystems within the pool using
>> raidz, the filesystems will have 25% less space available since this space
>> is used for parity data. So a filesystem using the whole pool will report
>> having 3.7 * 0.75 = 2.7TiB available which is in agreement with your
>> numbers. A raidz filesystem will always lose 1 disk worth of space and will
>> never report that space as available to you since it will always be occupied
>> with parity data.
>>
>> The pool on the other hand doesn't make a distinction, in this case
>> anyway, between user data and parity data so zpool will always report what's
>> actually unallocated on all your physical drives in the pool. For every GiB
>> you allocate in the filesystem you will allocate 1.33GiB in the pool since
>> that includes parity data. "zfs list" and "df -h" are your best friends to
>> find out how much space is available for your files. Don't bother about
>> "zpool list".
>>
>> Regards
>> Morgan
>> _______________________________________________
>> freebsd-questions at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>> To unsubscribe, send any mail to
>> "freebsd-questions-unsubscribe at freebsd.org"
>
>
> It gets even more hairy when you start adding in reservsions, quotas, and
> compression. Slap dedup on top of that and you get magically growing fs
> according to df 8)
>
>

Ahhh...yea. Thanks everyone! I didn't realize that zpool status would
show the raw space (so 4x1TB base10 is ~3.7TB base2), and not the
available space after the 4-1 RAIDZ usage consumes. So that all makes
sense, as well as with reservations, quotas and compression now
causing df to not really know what's going on. If I want to get the
actual values of available space vs used space, I should use zfs get
all <filesystem> and look at the properties there?

--Aaron