l2arc_feed_thread cpu utlization

Brendan Gregg brendan.gregg at joyent.com
Fri Feb 14 20:23:27 UTC 2014


G'Day Andriy,

Thanks for the patch. If most of the data is in one list (anyone have
statistics to confirm such a likelyhood? I know this happened a lot
pre-list-split), then I think this means we only scan that at 1/32nd of the
previous rate. It should solve the CPU issue, but could make warmup very
slow.

I think the feed algorithm needs to be rethought, although that can be done
as future work. I'm trying to think of what simple that can be done right
now to solve CPU usage and warmup rate.

Lets say we keep this change, but in l2arc_write_buffers we maintain an
extra copy of write_sz, say, list_write_sz, that is reset to zero for each
list. Then, when we reach headroom and choose to abort, we can check
list_write_sz and determine how fruitful the scanning has been so far. If
that's greater than a threshold, then keep scanning, up to the full
L2ARC_WRITE_SIZE for that list. That way, we've scanned only 1/32nd of the
previous length as a test, and only if that is fruitful enough do we keep
scanning.

Again, it probably needs to be rethought, but something like that may work
fine in the meantime.

Brendan



On Fri, Feb 14, 2014 at 3:52 AM, Andriy Gapon <avg at freebsd.org> wrote:

> on 19/12/2013 13:30 Andriy Gapon said the following:
> >
> > This is just a heads up, no patch yet.
> >
> > l2arc_feed_thread periodically wakes up and scans certain amount of ARC
> buffers
> > and writes eligible buffers to a cache device.
> > Number of scanned buffers is limited by a threshold on the amount of
> data in the
> > buffers seen.  The threshold is applied on a per buffer list basis.  In
> upstream
> > there are 4 relevant lists: (data, metadata) X (MFU, MRU).  In FreeBSD
> each of
> > the lists was subdivided into 16 lists.  This was done to reduce
> contention on
> > the locks that protect the lists.  But as a side effect
> l2arc_feed_thread can
> > scan 16 times more data (~ buffers).
> >
> > So, if you have a rather large ARC and L2ARC and your buffers tend to be
> > sufficiently small, then you could observe l2arc_feed_thread burning a
> > noticeable amount of CPU.  On some of our systems I observed it using up
> to 40%
> > of a single core.  Scaling back the threshold by factor of 16 makes CPU
> > utilization go down by the same factor.
> >
> > I plan to commit this change to FreeBSD ZFS code.
> > Any comments are welcome.
>
> Here is what I have in mind:
> https://github.com/avg-I/freebsd/compare/wip;hc;l2arc_feed_thread_scan_rate
>
> The calculations in the macro look somewhat ugly, but they should be
> correct :-)
>
> --
> Andriy Gapon
>
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>



-- 
Brendan Gregg, Joyent                      http://dtrace.org/blogs/brendan


More information about the freebsd-fs mailing list