ZFS: How to enable cache and logs.

[Jeremy Chadwick]

On Wed, May 11, 2011 at 07:25:52PM +1000, Danny Carroll wrote:
> I've been using ZFS for some time now and have never had an issued
> (except perhaps the issue of speed...)
> When v28 is taken into -STABLE I will most likely upgrade to v28 at that
> point.   Currently I am running v15 with v4 on disk.
> 
> When I move to v28 I will probably wish to enable a L2Arc and also
> perhaps dedicated log devices.
>
> I'm curious about a few things however.
> 
> 1. Can I remove either the L2 ARC or the log devices if things don't go
> as planned or if I need to free up some resources?

You can remove L2ARC ("cache") devices without impact, but you cannot
remove all log devices without the pool needing to be destroyed
(recreated).  Please keep reading for details of log devices.

L2ARC devices should primarily be something with extremely fast read
rates (e.g. SSDs).  USB1.x and 2.x memory sticks do not work well for
this purpose given protocol and bus speed limits + overhead.  (I only
mention them because people often think "Oh, USB flash would work great
for this!"  I disagree.)

Furthermore, something I found out on my own: the L2ARC is completely
lost in the case the system is cleanly rebooted.  This sometimes
surprises people (myself included) since L2ARC uses actual storage
devices; one might think the data is "restored" on reboot, but it isn't
(because the ARC ("layer 1") itself is lost on reboot, obviously).

The only way to see how much disk space a cache device is using -- to my
knowledge -- is via "zpool iostat -v".

> 2. What are the best practices for setting up these?   Would a geom
> mirror for the log device be the way to go.  Or can you just let ZFS
> mirror the log itself?

Let ZFS handle it.  There is no purpose (in my opinion) to added
complexity when ZFS can handle it itself.  The KISS concept applies
greatly here.

In the case of ZFS intent logs, you definitely want a mirror.  If you
have a single log device, loss of that device can/will result in full
data loss of the pool which makes use of the log device.

Furthermore, a log device is limited to a single pool; e.g. you cannot
use the same log device (e.g. ada6) on pool "foo" and pool "bar".  It's
one or the other.

You should read **all** of the data points listed below and pay close
attention to the details:

http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Separate_Log_Devices

> 3. What happens when one or both of the log devices fail.   Does ZFS
> come to a crashing halt and kill all the data?   Or does it simply
> complain that the ZIL is no longer active and continue on it's merry way?

See above.

> In short, what is the best way to set up these two features?

See the zpool(1) man page for details on how to make use of log devices.
Examples are provided, including mirroring of such devices.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |