An order of magnitude higher IOPS needed with ZFS than UFS

[Adam Nowacki]

On 2013-06-12 13:49, Jeremy Chadwick wrote:
> On Wed, Jun 12, 2013 at 06:40:32AM -0500, Mark Felder wrote:
>> On Tue, 11 Jun 2013 16:01:23 -0500, Attila Nagy <bra at fsn.hu> wrote:
>>
>>> BTW, the file systems are 77-78% full according to df (so ZFS
>>> holds more, because UFS is -m 8).
>>
>> ZFS write performance can begin to drop pretty badly when you get
>> around 80% full. I've not seen any benchmarks showing an improvement
>> with a very fast and large ZIL or tons of memory, but I'd expect
>> that would help significantly. Just note that you're right at the
>> edge where performance gets impacted.
>
> Mark, do you have any references for this?  I'd love to learn/read more
> about this engineering/design aspect (I won't say flaw, I'll just say
> aspect) to ZFS, as it's the first I've heard of it.
>
> The reason I ask: (respectfully, not judgementally) I'm worried you
> might be referring to something that has to do with SSDs and not ZFS,
> specifically SSD wear-levelling performing better with lots of free
> space (i.e. a small FTL map; TRIM helps with this immensely) -- where
> the performance hit tends to begin around the 70-80% mark.  (I can talk
> more about that if asked, but want to make sure the two things aren't
> being mistaken for one another)
>

So I went hunting for some evidence and created this:
http://tepeserwery.pl/nowak/fillingzfs.png

Columns are groups of sectors, new row is created every time a FLUSH 
command is sent to a disk. Percentage is the amount of filled space in 
the pool. Red means a write happened there, Pool is 1GB with writes of 
50MB between black lines.

It looks like past 80% there simply isn't enough continuous disk space 
and writes are becoming more and more random. For some unknown to me 
reason there is also a lot more flushing which certainly doesn't help 
for performance. There is also this odd hole left untouched by any 
write, reserved space of some sort?