RFC: Suggesting ZFS "best practices" in FreeBSD

[Kevin Day]

On Jan 22, 2013, at 9:12 AM, Warren Block <wblock at wonkity.com> wrote:
> 
> I would like to see guidelines for at least two common scenarios:
> 
> Multi-terabyte file server with multi-drive pool.

[…]

> The first is easy with the defaults, but particular tuning could be beneficial.  And would be a good place to talk about NFS on ZFS, usage of SSDs, and so on.


I run ftpmirror.your.org, which is a 72 x 3TB drive ZFS server. It's a very busy server. It currently houses the only off-site backup of all of the Wikimedia projects(121TB), a full FreeBSD FTP mirror(1T), a full CentOS mirror,  all of FreeBSD-Archive(1.5TB), FreeBSD-CVS, etc. It's usually running between 100 and 1500mbps of ethernet traffic in/out of it. There are usually around 15 FTP connections, 20-50 HTTP connections, 10 rsync connections and 1 or 2 CVS connections. 

The only changes we've made that are ZFS specific are atime=off and sync=disabled. Nothing we do uses atimes so disabling that cuts down on a ton of unnecessary writes. Disabling sync is okay here too - we're just mirroring stuff that's available elsewhere, so there's no threat of data loss. Other than some TCP tuning in sysctl.conf, this is running a totally stock kernel with no special settings. 

I've looked at using an SSD for meta-data only caching, but it appears that we've got far more than 256GB of metadata here that's being accessed regularly (nearly every file is being stat'ed when rsync runs) so I'm guessing it's not going to be incredibly effective unless I buy a seriously large SSD.

If you have any specific questions I'm happy to answer though.

-- Kevin