Best practices for ZFS setup for a strictly SSD based system?

[Alan Somers]

On Tue, Feb 9, 2016 at 8:54 AM, Patrick M. Hausen <hausen at punkt.de> wrote:
> Hi, all,
>
> while there is quite a bit of documentation on how to improve ZFS performance
> by using a combination of rotating disks and SSDs, I have not found much about
> an SSD only setup.
>
> We are planning to try a hosting server with 8 SATA SSDs with ZFS. Things I am
> not at all sure about:
>
> *       Does the recommended limit of 6 disks for a RAIDZ2 still
>         hold? 2x 4 disks is quite a bit of overhead, could I use all 8
>         in one vdev and get away with it?
>         (The maximum of 6 recommendation is in some old Sun doc)

Nah, you can go much higher.  This post describes the RAIDZ overhead.
The main penalty to larger stripes is lower IOPs.  Your RAIDZ array
will have the same read IOPs as a single SSD, no matter how large it
is.  So, for example, a pool of two RAIDZ stripes each containing 4+2
disks will have twice the IOPS as a pool containing one RAIDZ stripe
with 8+2 disks, and about the same storage overhead.

http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/

>
> *       Will e.g. MySQL still profit from residing on a mirror
>         instead of a RAIDZ2, even if all disks are SSDs?

Yes, because a mirrored vdev has as many read IOPs as all of its disks
combined.  So a RAID10 of SSDs will have many read IOPs indeed.

>
> *       Does a separate ZIL and/or ARC cache device still
>         make sense?

Usually no.  But it might make a difference if the ZIL or L2ARC
devices have different characteristics from the regular devices.  For
example, you might use medium speed MLC flash for your regular vdevs
and a very fast, small SLC device for the ZIL.  But I wouldn't do it
unless you thoroughly test it with your workload.

>
> Any pointers or direct help greatly appreciated. Or should I take this to freebsd-fs@?

Will MySQL access its files in fixed-size records?  If so, you can set
the recsize filesystem property accordingly.  If not, you should
probably leave recsize at the default.  If you profile MySQL's disk
accesses and determine that there is a dominant recordsize, then go
ahead and set ZFS's recsize to the next highest power of two.

As usual, disable atime.

>
> Thanks and best regards,
> Patrick
> --

-Alan