An order of magnitude higher IOPS needed with ZFS than UFS

[Robert Schulze]

Hi,

Am 12.06.2013 13:49, schrieb Jeremy Chadwick:
> On Wed, Jun 12, 2013 at 06:40:32AM -0500, Mark Felder wrote:
>> On Tue, 11 Jun 2013 16:01:23 -0500, Attila Nagy <bra at fsn.hu> wrote:
>>
>> ZFS write performance can begin to drop pretty badly when you get
>> around 80% full. I've not seen any benchmarks showing an improvement
>> with a very fast and large ZIL or tons of memory, but I'd expect
>> that would help significantly. Just note that you're right at the
>> edge where performance gets impacted.
>
> Mark, do you have any references for this?  I'd love to learn/read more
> about this engineering/design aspect (I won't say flaw, I'll just say
> aspect) to ZFS, as it's the first I've heard of it.

this is even true when getting near a quota limit on a zfs, although 
there are e.g. 10/16 TB free in the pool.

Just create a filesystem and set quota=1G, then do sequential 
invocations of dd to fill the fs with 100M files. You will see a sharp 
slowdown when the last twenty files are beeing created.

Here are the results from the following short test:

for i in `jot - 0 99`
     do
     dd if=/dev/zero of=/pool/quota-test/10M.$i bs=1M count=10
     done

0..80:	< 0.4 s
80	0.27 s
81	0.77 s
82	0.50 s
83	0.51 s
84	0.22 s
85	0.87 s
86	0.52 s
87	1.13 s
88	0.91 s
90	0.39 s
91	1.04 s
92	0.80 s
93	1.94 s
94	1.27 s
95	1.36 s
96	1.76 s
97	2.13 s
98	3.28 s
99	4.07 s

of course, there are some small values beyond 80% utilisation, but I 
think the trend is clearly visible.

In my opinion, hitting a quota limit should not give these results 
unless enough free physical disk space is available in the pool. This is 
a bug or a design flaw and creating serious problems when exporting 
quota'ed zfs over nfs.

with kind regards,
Robert Schulze

-- 
/7\ bytecamp GmbH
Geschwister-Scholl-Str. 10, 14776 Brandenburg a.d. Havel
HRB15752, Amtsgericht Potsdam, Geschaeftsfuehrer:
Bjoern Barnekow, Frank Rosenbaum, Sirko Zidlewitz
tel +49 3381 79637-0 werktags 10-12,13-17 Uhr, fax +49 3381 79637-20
mail rs at bytecamp.net, web http://bytecamp.net/