ZFS L2ARC - incorrect size and abnormal system load on r255173

[Vitalij Satanivskij]

Hello 

Bad news, to day two servers with applayed patches get L2ARC degraded 

L2 ARC Summary: (DEGRADED)
        Passed Headroom:                        4.17m
        Tried Lock Failures:                    635.53m
        IO In Progress:                         41.89k
        Low Memory Aborts:                      8
        Free on Write:                          1.03m
        Writes While Full:                      12.95k
        R/W Clashes:                            405.05k
        Bad Checksums:                          362.19k
        IO Errors:                              45.60k
        SPA Mismatch:                           526.37m

L2 ARC Size: (Adaptive)                         391.03  GiB
        Header Size:                    0.59%   2.30    GiB

So looks like problem not disapered ^( 




Vitalij Satanivskij wrote:
VS> 
VS> First of all Thank you for help.
VS> 
VS> As for high load on system, looks like problems with l2arc have litle impact on load comparatively to another just now 
VS> 
VS> not fully classifed things.
VS> 
VS> Looks like ower internal software and libs that it use didn't like new VMEM subsystem, at last 
VS> system behavior complitely diferent from 6 month older CURRENT. 
VS> 
VS> So for now none problem's with l2arc errors.
VS> 
VS> Will try to understand reason of load and fix or at last ask for help again ^).
VS> 
VS> 
VS> 
VS> 
VS> Steven Hartland wrote:
VS> SH> First off I just wanted to clarify that you don't need to compression on
VS> SH> dataset for L2ARC to use LZ4 compression, it does this by default as is
VS> SH> not currently configurable.
VS> SH> 
VS> SH> Next up I believe we've found the cause of this high load and I've just
VS> SH> committed the fix to head:
VS> SH> http://svnweb.freebsd.org/base?view=revision&sortby=file&revision=256889
VS> SH> 
VS> SH> Thanks to Vitalij for testing :)
VS> SH> 
VS> SH> Dmitriy if you could test on your side too that would be appreciated.
VS> SH> 
VS> SH>     Regards
VS> SH>     Steve
VS> SH> 
VS> SH> ----- Original Message ----- 
VS> SH> From: "Vitalij Satanivskij" <satan at ukr.net>
VS> SH> To: "Allan Jude" <freebsd at allanjude.com>
VS> SH> Cc: <freebsd-current at freebsd.org>
VS> SH> Sent: Thursday, October 10, 2013 6:03 PM
VS> SH> Subject: Re: ZFS L2ARC - incorrect size and abnormal system load on r255173
VS> SH> 
VS> SH> 
VS> SH> > AJ> Some background on L2ARC compression for you:
VS> SH> > AJ> 
VS> SH> > AJ> http://wiki.illumos.org/display/illumos/L2ARC+Compression
VS> SH> > 
VS> SH> > I'm alredy see it.
VS> SH> > 
VS> SH> > 
VS> SH> > 
VS> SH> > AJ> http://svnweb.freebsd.org/base?view=revision&revision=251478
VS> SH> > AJ> 
VS> SH> > AJ> Are you sure that compression on pool/zfs is off? it would normally
VS> SH> > AJ> inherit from the parent, so double check with: zfs get compression pool/zfs
VS> SH> > 
VS> SH> > Yes, compression turned off on pool/zfs, it's was may time rechecked.
VS> SH> > 
VS> SH> > 
VS> SH> > 
VS> SH> > AJ> Is the data on pool/zfs related to the data on the root pool? if
VS> SH> > AJ> pool/zfs were a clone, and the data is actually used in both places, the
VS> SH> > AJ> newer 'single copy ARC' feature may come in to play:
VS> SH> > AJ> https://www.illumos.org/issues/3145
VS> SH> > 
VS> SH> > No, both pool and pool/zfs have diferent type of data, pool/zfs was created as new empty zfs (zfs create pool/zfs)
VS> SH> > 
VS> SH> > and data was writed to it from another server. 
VS> SH> > 
VS> SH> > 
VS> SH> > Right now one machine work fine with l2arc. This machine without patch for corecting ashift on cache devices.
VS> SH> > 
VS> SH> > At last 3 day's working with zero errors. Another servers with same config similar data, load and so on after 2 day 
VS> SH> > work began report abouy errors.
VS> SH> > 
VS> SH> > 
VS> SH> > AJ> 
VS> SH> > AJ> 
VS> SH> > AJ> 
VS> SH> > AJ> -- 
VS> SH> > AJ> Allan Jude
VS> SH> > AJ> 
VS> SH> > AJ> _______________________________________________
VS> SH> > AJ> freebsd-current at freebsd.org mailing list
VS> SH> > AJ> http://lists.freebsd.org/mailman/listinfo/freebsd-current
VS> SH> > AJ> To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"
VS> SH> > _______________________________________________
VS> SH> > freebsd-current at freebsd.org mailing list
VS> SH> > http://lists.freebsd.org/mailman/listinfo/freebsd-current
VS> SH> > To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"
VS> SH> >
VS> SH> 
VS> SH> ================================================
VS> SH> This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 
VS> SH> 
VS> SH> In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
VS> SH> or return the E.mail to postmaster at multiplay.co.uk.
VS> SH> 
VS> SH> _______________________________________________
VS> SH> freebsd-current at freebsd.org mailing list
VS> SH> http://lists.freebsd.org/mailman/listinfo/freebsd-current
VS> SH> To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"
VS> _______________________________________________
VS> freebsd-current at freebsd.org mailing list
VS> http://lists.freebsd.org/mailman/listinfo/freebsd-current
VS> To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"