zfs L2ARC wrong alloc/free size and DEGRADED status

[Таболин Юрий]

Hi All.

I have server FreeBSD 10.1-RELEASE r274401 GENERIC kernel amd64 with 2 
zfs pools. There are two Intel 480Gb SSD disks in server, used like ZIL 
(mirror 4GB per pool) and L2ARC (stripe 75+75 GB per pool). Some days 
ago I noticed wrong L2ARC size pool1 in zpool iostat -v, later I seen 
same wrong size in pool2. Now it looks like:

poolallocfreereadwritereadwrite

------------------------------------------------------------------

pool113,0T34,3T453,56K3,93M51,9M

raidz313,0T34,3T453,51K3,93M47,1M

multipath/pd01--3197311K4,96M

multipath/pd02--3197311K4,96M

multipath/pd03--3197311K4,96M

multipath/pd04--3197311K4,96M

multipath/pd05--3197311K4,96M

multipath/pd06--3197311K4,96M

multipath/pd07--3197311K4,96M

multipath/pd08--3197311K4,96M

multipath/pd09--3197311K4,96M

multipath/pd10--3197311K4,96M

multipath/pd11--3197311K4,96M

multipath/pd12--3197311K4,96M

multipath/pd13--3197311K4,96M

logs------

mirror812K3,97G04504,83M

diskid/DISK-CVWL435200Y1480QGNp1--04544,83M

diskid/DISK-CVWL4353000F480QGNp1--04544,83M

cache------

diskid/DISK-CVWL435200Y1480QGNp4371G16,0E427163K3,16M

diskid/DISK-CVWL4353000F480QGNp4441G16,0E825145K2,94M

------------------------------------------------------------------

pool210,2T37,0T811,36K9,82M80,2M

raidz310,2T37,0T818709,82M45,9M

multipath/pd14--2182903K4,67M

multipath/pd15--2182903K4,67M

multipath/pd16--2182903K4,67M

multipath/pd17--2182903K4,67M

multipath/pd18--2182904K4,67M

multipath/pd19--2182903K4,67M

multipath/pd20--2182903K4,67M

multipath/pd21--2182903K4,67M

multipath/pd22--2182903K4,67M

multipath/pd23--2182903K4,67M

multipath/pd24--2182903K4,67M

multipath/pd25--2182903K4,67M

multipath/pd26--2182903K4,67M

logs------

mirror238M3,74G0525034,3M

diskid/DISK-CVWL435200Y1480QGNp2--0525434,3M

diskid/DISK-CVWL4353000F480QGNp2--0525434,3M

cache------

diskid/DISK-CVWL435200Y1480QGNp5207G16,0E12145,1K2,56M

diskid/DISK-CVWL4353000F480QGNp5203G16,0E22194,6K2,60M


Values “371G16,0E” are abnormal, real aloc size is 75G.

After that I looked zfs-stat -L and seen DEGRADED L2ARC and too big 
L2ARC size:

L2 ARC Summary: (DEGRADED)

Passed Headroom:6.05m

Tried Lock Failures:22.36m

IO In Progress:2.75k

Low Memory Aborts:2.86k

Free on Write:5.48m

Writes While Full:339.48k

R/W Clashes:2.07k

Bad Checksums:211.52k

IO Errors:101.41k

SPA Mismatch:3.16b

L2 ARC Size: (Adaptive)1.27TiB

Header Size:1.42%18.56GiB

kstat.zfs.misc.arcstats.l2_io_error: 101531

kstat.zfs.misc.arcstats.l2_cksum_bad: 211782

smartctl told that both SSD is fine, without any IO errors. I found same 
issues which described about L2ARC compression:

http://forums.freebsd.org/threads/l2arc-degraded.47540/

http://lists.freebsd.org/pipermail/freebsd-current/2013-October/045088.html

My problem looks like same bug. Thanks for any help!

-- 
Best regards,

Tabolin Yuriy
System administrator
Speech Technology Center