Optimizing performance with SLOG/L2ARC

[PK1048]

Lots of good comments already posted, more inline below...

On Aug 18, 2015, at 20:28, javocado <javocado at gmail.com> wrote:

> I've been trying to optimize and enhance my ZFS filesystem performance
> (running FreeBSD 8.3amd) which has been sluggish at times. Thus far I have
> added RAM (256GB) and I've added an SLOG (SSD mirror). The RAM seems to
> have helped a bit, but not sure if the SLOG was of much help. My vdev is
> decently busy, with writes and reads averaging at 100 per second with
> spikes as high as 500.

What is the system doing _besides_ being a file server ? From the arcstats, the ARC is not getting all of the 256 GB RAM, what else is running ?

What is the pool layout (zpool status)… From a strictly a performance standpoint, each vdev gives you ONE drives worth of WRITE performance. It does not much matter if the vdev is a mirror or RAIDz<n>. So a pool consisting of one vdev which is a 12 drive RAIDz2 will have much slower write performance that a pool consisting of 3 vdevs each of which is a 4 drive RAIDz2 … Yes, you are balancing off capacity vs. performance vs. reliability.

<snip>

> 1. based on the output above, I believe a larger ARC may not necessarily
> benefit me at this point. True?

The arcstats look good, are you having issues with reads or writes ? If reads, then you need to now look at your disk layout. If writes, then are they sync or async, if sync then a ZIL can help, if async, then more ARC can help and you need to look at your disk layout.

> 2. Is more (L2)ARC always better?

ARC is _aways_ better than L2ARC. Adding L2ARC also consumes ARC space with L2ARC metadata, and that is ARC space that cannot now be used to cache file metadata and data. So adding L2ARC (which is slower than RAM based ARC) _may_ actually make a system slower.

> 3. I know it's a good idea to mirror the SLOG (and I have). Do I understand
> correctly that I do not need to mirror the L2ARC since it's just a read
> cache, nothing to lose if the SSD goes down?

L2ARC is just read cache, so loosing it causes no data loss. You _may_ lose performance _if_ the L2ARC was increasing your performance.

> 4. Is there a better way than looking at zpool iostat -v to determine the
> SLOG utilization and usefulness?

The ZIL/SLOG is used to cache SYNC write operations (if you test with iozone, use the -o option to force sync writes). Using iostat -x you can see the activity to the SSDs, both write and read. This can tell you how much you are using them. I am not aware of any way to track how FULL your ZIL/SLOG device is.

Please note that, depending on your workload, an SSD may _not_ be any faster than a HDD. I am in the process of rebuilding a file server that exhibited poor NFS SYNC write performance. Yet it had a mirrored pair of SSDs. Unfortunately, those SSDs had _worse_ write performance than an HDD for small (4 KB) writes. Based on recommendations from the OpenZFS list I have a pair of Intel 3710 SSDs coming in to try, they are supposed to have much better write performance (at all block sizes) and much better reliability long term (10x full disk writes per day for 5 years). I’ll know more once they arrive and I can test with them.

Someone commented on the size of the ZIL/SLOG… it needs to hold all of the write data that arrives between TXG commits, which happen at least every 5 seconds (it used to be 30 seconds, but that scared too many people :-). SU a sync write arrives and it _must_ be committed to permanent storage, so ZFS writes it to the ZFS Intent Log (ZIL) which may or may not be a separate device (vdev). When the TXG that contains that data is committed to the pool itself the data can be flushed from the ZIL. If your source of sync writes is network shares, and you have a 1 Gbps link, then your maximum ZIL will be 5 seconds x 1 Gbps or 5 Gigabits.

> I'd like to test-see if adding L2ARC yields any performance boost. Since
> SLOG isn't doing much for me, I'm thinking I could easily repurpose my SLOG
> into an L2ARC.

If a ZIL/SLOG device is not adding any performance, then either your workload is NOT sync write bound -or- your ZIL/SLOG device is no faster than your native pool performance. If the ZIL/SLOG device is not very fast, then reusing it as an L2ARC -may- actually make the system slower.

> 
> Questions:
> 
> 5. In testing, it seemed fine to remove the SLOG from a live/running system
> (zpool remove pool mirror-3). Is this in fact a safe thing to do to a
> live/running system? ZFS knows that it should flush the ZIL, then remove
> the device? Is it better or necessary to shut down the system and remove
> the SLOG in "read only" mode?

There was a time when you _could_not_ remove a LOG device once added to a pool. A specific revision of the zpool added the ability to remove a LOG device. As long as your zpool is a late enough version. zpool upgrade -v will tell you what version / features you have. It looks like version 19 added log removal, while 7 added the log itself :-) So if your system and zpool are version 19 or later (and I think all FreeBSD ZFS code is) then it is completely safe to remove a log device and repurpose it.

> 6. Am I missing something about how the SLOG and L2ARC play together that I
> would miss by running my proposed test. i.e. if I take down the SLOG and
> repurpose as an L2ARC might I be shooting myself in the foot cause the SLOG
> and L2ARC combo is much more powerful than the L2ARC alone (or SLOG alone)?
> My hope here is to see if the L2ARC improves performance, after which I
> will proceed with buying the SSD(s) for both the SLOG and L2ARC.

It all depends on your workload… Sync Writes are helped by the ZIL/SLOG, it does nothing for reads. ARC helps both async writes and all reads. L2ARC helps all reads.