ZFS performance degradation over time

[Ivan Voras]

Garrett Moore wrote:
> No, I haven't isolated the cause to only be uptime related. In my original
> email I mentioned that "as suggested by someone in the thread, it's probably
> not directly related to system uptime, but instead related to usage - the
> more usage, the worse the performance."

Ok, noted :)

> I've been starting my system with different combinations of applications
> running to see what access patterns cause the most slowdown. So far, I don't
> have enough data to give anything concrete.
> 
> This weekend I'll try some tests such as the one you describe, and see what
> happens. I have a strong suspicion that rTorrent is to blame, since I
> haven't seen major slowdowns in the last few days with rTorrent not running.
> rTorrent preallocates the space needed for the file download (and I'm
> downloading large 4GB+ files using it), and then writes to them in an
> unpredictable pattern, so maybe ZFS doesn't like being touched this way?

This is why I thought the test with a single large file written before 
"slowdown" and read after could be helpful.

It is true that ZFS in theory doesn't do very well with random writes of 
any kind - the kind that torrent clients do should actually be the worst 
case for ZFS, *but*, this very much depends on the actual workload.

This is because, simplified, ZFS *always* appends data to files and then 
does a type of pointer gymnastics to update the actual file logical 
"flow" from the beginning to the end. For example, doing thousands of 
random small writes where each of them is logically located at another 
position in the file (what torrent clients do) will actually write them 
all together in some not entirely predictable order - then when the file 
is read in its logical/natural order, the drives' heads will have to 
seek much more often to pick up the pieces. Thus, a 700 MB file could 
end up in a thousand fragments. There are some very good reasons why ZFS 
does things that way but it makes it sensitive to performance issues in 
the described scenario. Preallocating space will not help.

Surprisingly, this doesn't appear to harm databases much - at least the 
usual kinds - because they tend to avoid sequential reads by using 
tricks like indexing and so are used to paying the price for disk head 
seeking.

Of course, SSDs would not notice anything unusual with this mode of 
operation.