OpenZFS and L2ARC

[Ryan Moeller]

On Tue, Sep 8, 2020 at 3:43 AM Stefan Esser <se at freebsd.org> wrote:
>
> OpenZFS seems to work quite well for me, in general, but I have
> questions regarding the L2ARC statistics.
>
> The system uses a 3 * 6 TB raidz1 (plus further ZFS volumes that
> are not relevant here, since without level 2 ARC) and an 1 TB M.2
> SSD with a 256 GB partition for the L2ARC (and most of it currently
> unused, else).
>
> The L2ARC seems to have filled to the limit of 256 GB, but after
> several reboots, sysctl reports a L2ARC size of nearly twice the
> allocated space:
>
> kstat.zfs.misc.arcstats.l2_size: 534620858880
>
> That is 497 GiB, and might be possible with a lz4 compression
> factor of 2 - if the value reported is not the space allocated,
> but the actual (uncompressed) data held by the L2ARC.
>
>
> The sysutils/zfs-stats port reports the following values for
> this system, BTW:
>
> ------------------------------------------------------------------------
> ZFS Subsystem Report                            Tue Sep  8 09:02:46 2020
> ------------------------------------------------------------------------
>
> L2 ARC Summary: (HEALTHY)
>         Passed Headroom:                        0
>         Tried Lock Failures:                    0
>         IO In Progress:                         0
>         Low Memory Aborts:                      7
>         Free on Write:                          123
>         Writes While Full:                      0
>         R/W Clashes:                            0
>         Bad Checksums:                          0
>         IO Errors:                              0
>         SPA Mismatch:                           0
>
> L2 ARC Size: (Adaptive)                         497.91  GiB
>         Header Size:                    0.11%   558.83  MiB
>
> L2 ARC Evicts:
>         Lock Retries:                           6
>         Upon Reading:                           0
>
> L2 ARC Breakdown:                               5.75    m
>         Hit Ratio:                      81.94%  4.71    m
>         Miss Ratio:                     18.06%  1.04    m
>         Feeds:                                  235.04  k
>
> L2 ARC Buffer:
>         Bytes Scanned:                          0       Bytes
>         Buffer Iterations:                      0
>         List Iterations:                        0
>         NULL List Iterations:                   0
>
> L2 ARC Writes:
>         Writes Sent:                    100.00% 22.67   k
>
> ------------------------------------------------------------------------
>
> With the FreeBSD ZFS (without persistent L2ARC) I never got more
> than 20% hit ratio on the L2ARC between reboots.
>
> Quite a number of sysctl variable names have changed, and the port
> needs to be adapted to the new names (therefore there are lots of 0
> values in the -L output).
>
> The following names used by zfs-stats do not exist in OpenZFS:
>
> kstat.zfs.misc.arcstats.recycle_miss
> kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned
> kstat.zfs.misc.arcstats.l2_write_buffer_iter
> kstat.zfs.misc.arcstats.l2_write_buffer_list_iter
> kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter
> kstat.zfs.misc.arcstats.l2_write_full
> kstat.zfs.misc.arcstats.l2_write_in_l2
> kstat.zfs.misc.arcstats.l2_write_io_in_progress
> kstat.zfs.misc.arcstats.l2_write_not_cacheable
> kstat.zfs.misc.arcstats.l2_write_passed_headroom
> kstat.zfs.misc.arcstats.l2_write_pios
> kstat.zfs.misc.arcstats.l2_write_spa_mismatch
> kstat.zfs.misc.arcstats.l2_write_trylock_fail
> kstat.zfs.misc.arcstats.l2_writes_hdr_miss
> vfs.zfs.vdev.cache.size
>
> The existence of vfs.zfs.vdev.cache.size vs vfs.zfs.vdev.cache_size
> can be used to detect OpenZFS, and is easily fixed.
>
> But the above listed L2ARC values seem to have been removed from or
> have never existed in OpenZFS, and I did not find any substitutes.
>
> Are there any plans to re-create them in OpenZFS on FreeBSD or are
> they gone for good?
>
> I'd like to update the zfs-stats port for compatibilíty with OpenZFS ...

Improved L2ARC stats is something we would like to look into, yes.
There is currently one PR open to add a few more arcstats for L2:

* evict_l2_eligible_mfu
* evict_l2_eligible_mru
* l2_prefetch_asize
* l2_mru_asize
* l2_mfu_asize
* l2_bufc_data_asize
* l2_bufc_metadata_asize

Adding the other L2ARC arcstats from FreeBSD would be a welcome
change, as well. It's *somewhere* on the todo list.


--
Ryan Moeller
iXsystems, Inc.
OS Developer
Email: ryan at iXsystems.com