L2ARC and ZIL on same SSD?

[Daniel Kalchev]

On 20.05.13 17:53, Josh Beard wrote:
> Hello,
>
> I have a file server running 9.1-RELEASE with 24 GB of RAM and a 6 TB zpool
> spread across 12 500GB disks in two raidz1 and 2 spares.  This server
> houses hundreds of home directories for end-users and stays pretty busy
> with both read and writes.
>
> I'm planning to add two SSDs soon (128 GB each) to help performance and was
> planning to mirror them. I was curious if having the L2ARC and ZIL on the
> same disks (mirrored) is okay.  With some searching, I haven't seen any
> "official" word on this.  I've seen some folks say not to and others
> recommend it.  Adding a second set of SSDs isn't desirable, as I'll start
> cutting into my available storage capacity (16 bay system).
>
> Any advice or experience with this?
>


Indeed, there is no official position on how SSDs should be used with 
ZFS, because SSDs are so much different from each other. Here is what 
you should consider, however:

There is no need to mirror the L2ARC. On failure, it is simply ignored. 
Therefore, for L2ARC in general you put as large SSD as you find, that 
has good read rate. You might want to have this on a separate 
controller, which provides the highest bandwidth.

There is suggestion to mirror the ZIL, in order to avoid pool damage, 
but in post-v28 pools, the ZIL can be removed and replaced on the fly. 
Since the ZIL is only ever read after crash, it all boils down what your 
recovery scenario is. If you can tolerate the loss of most recent data, 
you really do not need to mirror ZIL.

In any case, it is a bad idea to mirror the SSDs before giving them to 
ZFS. Let ZFS manage it's redundancy, by giving it either the raw devices 
or raw slices/partittions.

You also need to start thinking of SSDs in ZFS as "expense". You are 
wearing it during use and at some point you will have to replace the 
SSD. That is, you trade SSD wear for performance. L2ARC and ZIL have 
quite different wearing characteristics.

In theory, ZIL would work best if it is either an high endurance SLC 
flash drive, or if you use only small portion of it for writes. For 
example, use only 8GB out of an 30GB (or larger) SSD for ZIL.

L2ARC, if sufficiently large would wear much less, because it will be 
mostly used for reads. But, if your L2ARC is smaller than required -- it 
will be overwriten many many times and thus wear faster. In both cases 
you can monitor the "Media Wearout Indicator" of SMART to see how much 
life you have left.

So considering especially the "expense" concept, you may want to have 
one higher performance but small SSD for ZIL and another lower 
performance but cheap SSD for L2ARC. If your SSD has sufficient 
read/write performance, you might use slices of the same SSD for ZIL and 
L2ARC. But consider consumer grade "performance" SSD wear faster. One 
good things is that as SSDs wear, larger and cheaper SSDs will be 
available :)

I don't know how much this helps -- you need to put it into perspective 
for your workload and other constraints.

Daniel