ZFS performance as the FS fills up?

[Jeremy Chadwick]

On Tue, Mar 08, 2011 at 12:26:49PM +0100, Patrick M. Hausen wrote:
> we use a big JBOD and ZFS with raidz2 as the target
> for our nightly Amanda backups.
> 
> I already suspected that the fact that the FS was > 90% full might
> be the cause of our backup performance continously decreasing.
> 
> I just added another vdev - 6 disks of 750 GB each, raidz2 and the
> FS usage is back to 71% currently. This was while backups were
> running and write performance instantly skyrocketed compared to
> the values before.
> 
> So, is it possible to name a reasonable amount of free space to
> keep on a raidz2 volume? On last year's EuroBSDCon I got
> the impression that with recent (RELENG_8) ZFS merges
> I could get away with using around 90%.

I'm in no way attempting to dissuade you from your efforts to figure out
a good number for utilisation, but when I hear of disks -- no matter how
many -- being 90% full, I immediately conclude performance is going to
suck simply because the outer "tracks" on a disk contains more sectors
than the inner "tracks".  This is the reason for performance degradation
as the seek offset increases, resulting in graphs like this:

http://img171.imageshack.us/img171/4776/1tb2.png

Given this info, it shouldn't come as too much of a surprise that adding
another vdev (effectively adding more disks to the pool) greatly helps
in relieving this issue.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |