[Bug 195746] zfs L2ARC wrong alloc/free size

bugzilla-noreply at freebsd.org bugzilla-noreply at freebsd.org
Fri Feb 13 16:04:45 UTC 2015


https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195746

John K. Gates <john at infoassurancepro.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |john at infoassurancepro.com

--- Comment #1 from John K. Gates <john at infoassurancepro.com> ---
I have been discussing this issue here:

https://forums.freebsd.org/threads/l2arc-degraded.47540/

Checksum and IO errors appear after an L2ARC device fills completely with cache
data on any release of FreeBSD after L2ARC compression
(http://wiki.illumos.org/display/illumos/L2ARC+Compression) was enabled
(9.3-RELEASE and later, and 10.0-RELEASE and later).

The following is from an 10.1-RELEASE-p5 (svn rev 278678) running a generic
kernel.  /etc/make.conf contains only "CPUTYPE?=core2".  ZFS pools are v28
(created under 9.2-RELEASE) and have NOT been updated for feature flags.

"zfs-stats -L" shows the L2ARC "DEGRADED" with numerous I/O and checksum
errors:

root at cadence:/ # zfs-stats -L

------------------------------------------------------------------------
ZFS Subsystem Report                            Fri Feb 13 10:36:49 2015
------------------------------------------------------------------------

L2 ARC Summary: (DEGRADED)
        Passed Headroom:                        30.28m
        Tried Lock Failures:                    24.83m
        IO In Progress:                         247
        Low Memory Aborts:                      103
        Free on Write:                          54.97k
        Writes While Full:                      10.62k
        R/W Clashes:                            562
        Bad Checksums:                          1.29m
        IO Errors:                              128.28k
        SPA Mismatch:                           48.53b

L2 ARC Size: (Adaptive)                         33.51   GiB
        Header Size:                    2.24%   768.85  MiB

L2 ARC Evicts:
        Lock Retries:                           18
        Upon Reading:                           0

L2 ARC Breakdown:                               35.47m
        Hit Ratio:                      26.64%  9.45m
        Miss Ratio:                     73.36%  26.02m
        Feeds:                                  568.79k

L2 ARC Buffer:
        Bytes Scanned:                          530.43  TiB
        Buffer Iterations:                      568.79k
        List Iterations:                        36.17m
        NULL List Iterations:                   974.71k

L2 ARC Writes:
        Writes Sent:                    100.00% 136.10k

------------------------------------------------------------------------

Kernel variables showing compression is working on L2ARC, and there are I/O and
checksum errors:

root at cadence:/ # sysctl kstat.zfs.misc.arcstats.l2_compress_successes
kstat.zfs.misc.arcstats.l2_compress_zeros
kstat.zfs.misc.arcstats.l2_compress_failures
kstat.zfs.misc.arcstats.l2_cksum_bad kstat.zfs.misc.arcstats.l2_io_error
kstat.zfs.misc.arcstats.l2_compress_successes: 1353514
kstat.zfs.misc.arcstats.l2_compress_zeros: 29
kstat.zfs.misc.arcstats.l2_compress_failures: 4985
kstat.zfs.misc.arcstats.l2_cksum_bad: 1290021
kstat.zfs.misc.arcstats.l2_io_error: 128275

Slight variations of this problem have been reported in numerous instances on
FreeBSD, FreeNAS and PC-BSD related forums and mailing lists, and is usually
dismissed as a hardware problem:

https://bugs.freenas.org/issues/5347
https://forums.freenas.org/index.php?threads/l2-arc-summary-degraded.19256/
https://bugs.pcbsd.org/issues/3418
http://svnweb.freebsd.org/base?view=revision&sortby=file&revision=256889
http://lists.freebsd.org/pipermail/freebsd-current/2013-October/045088.html
http://lists.freebsd.org/pipermail/freebsd-bugs/2014-November/059261.html
http://lists.freebsd.org/pipermail/freebsd-bugs/2014-December/059376.html
http://lists.freebsd.org/pipermail/freebsd-fs/2014-October/020242.html

However, I've been able to easily duplicate this problem on two different sets
of high quality, reliable hardware (Dell PowerEdge, Intel SSD) that otherwise
tests perfectly.

To duplicate, simply create a zfs pool with a small L2ARC device and exercise
the pool with random I/O until the L2ARC fills.

-- 
You are receiving this mail because:
You are the assignee for the bug.


More information about the freebsd-bugs mailing list