An order of magnitude higher IOPS needed with ZFS than UFS

[Matthew Ahrens]

The reason for this behavior is that it's hard to know how much space
things will take up on disk until we get to writing them out.  For example,
because compression changes the space usage, and we don't compress the data
until we write it out. (And all metadata, (e.g. indirect blocks) are
compressed, and count towards quotas.)

I think it would be reasonable to implement something like
"quota_enforcement=delayed", which would cause regular quotas,
reservations, refquotas, and refreservations to behave like userquota at ...
quotas, in that they can be exceeded by a small amount (a few seconds worth
of change), but they do not impact performance.

--matt


On Thu, Jun 13, 2013 at 1:28 AM, Robert Schulze <rs at bytecamp.net> wrote:

> Hi,
>
> Am 12.06.2013 13:49, schrieb Jeremy Chadwick:
>
>> On Wed, Jun 12, 2013 at 06:40:32AM -0500, Mark Felder wrote:
>>
>>> On Tue, 11 Jun 2013 16:01:23 -0500, Attila Nagy <bra at fsn.hu> wrote:
>>>
>>> ZFS write performance can begin to drop pretty badly when you get
>>> around 80% full. I've not seen any benchmarks showing an improvement
>>> with a very fast and large ZIL or tons of memory, but I'd expect
>>> that would help significantly. Just note that you're right at the
>>> edge where performance gets impacted.
>>>
>>
>> Mark, do you have any references for this?  I'd love to learn/read more
>> about this engineering/design aspect (I won't say flaw, I'll just say
>> aspect) to ZFS, as it's the first I've heard of it.
>>
>
> this is even true when getting near a quota limit on a zfs, although there
> are e.g. 10/16 TB free in the pool.
>
> Just create a filesystem and set quota=1G, then do sequential invocations
> of dd to fill the fs with 100M files. You will see a sharp slowdown when
> the last twenty files are beeing created.
>
> Here are the results from the following short test:
>
> for i in `jot - 0 99`
>     do
>     dd if=/dev/zero of=/pool/quota-test/10M.$i bs=1M count=10
>     done
>
> 0..80:  < 0.4 s
> 80      0.27 s
> 81      0.77 s
> 82      0.50 s
> 83      0.51 s
> 84      0.22 s
> 85      0.87 s
> 86      0.52 s
> 87      1.13 s
> 88      0.91 s
> 90      0.39 s
> 91      1.04 s
> 92      0.80 s
> 93      1.94 s
> 94      1.27 s
> 95      1.36 s
> 96      1.76 s
> 97      2.13 s
> 98      3.28 s
> 99      4.07 s
>
> of course, there are some small values beyond 80% utilisation, but I think
> the trend is clearly visible.
>
> In my opinion, hitting a quota limit should not give these results unless
> enough free physical disk space is available in the pool. This is a bug or
> a design flaw and creating serious problems when exporting quota'ed zfs
> over nfs.
>
> with kind regards,
> Robert Schulze
>
> --
> /7\ bytecamp GmbH
> Geschwister-Scholl-Str. 10, 14776 Brandenburg a.d. Havel
> HRB15752, Amtsgericht Potsdam, Geschaeftsfuehrer:
> Bjoern Barnekow, Frank Rosenbaum, Sirko Zidlewitz
> tel +49 3381 79637-0 werktags 10-12,13-17 Uhr, fax +49 3381 79637-20
> mail rs at bytecamp.net, web http://bytecamp.net/
>
> ______________________________**_________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/**mailman/listinfo/freebsd-fs<http://lists.freebsd.org/mailman/listinfo/freebsd-fs>
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@**freebsd.org<freebsd-fs-unsubscribe at freebsd.org>
> "
>