l2arc_feed_thread cpu utlization

[Brendan Gregg]

G'Day Andriy,

On Sat, Feb 15, 2014 at 3:58 AM, Andriy Gapon <avg at freebsd.org> wrote:

> on 14/02/2014 22:23 Brendan Gregg said the following:
> > G'Day Andriy,
> >
> > Thanks for the patch. If most of the data is in one list (anyone have
> statistics
> > to confirm such a likelyhood? I know this happened a lot
> pre-list-split), then I
> > think this means we only scan that at 1/32nd of the previous rate. It
> should
> > solve the CPU issue, but could make warmup very slow.
>
> Brendan,
>
> I do not have any stats, but I think that the data should be spread more
> or less
> evenly between the lists.  I mean the 16 sub-lists for data and 16
> sub-lists for
> metadata.  First, a list is picked up based on hash and that _should_
> produce
> more or less even distribution.  Second, if the hash funciton is not good
> enough
> then whole list splitting is pointless.
> In either case this was just a quick hack on my part.
>

Ah, I'm sorry, I should have read more of the code earlier; I had assumed
the split
algorithm was something else, and I'm wrong. It should be even.

So, based on get_buf_info(), I think we can DTrace how buf_hash() is mapped
to
the lists to get an idea of the distribution. Eg (on illumos, which has the
same
buf_hash() code):

# dtrace -n 'fbt::buf_hash:return { @ = lquantize(arg1 & (32 - 1), 0, 32,
1); } tick-30s { exit(0); }'
dtrace: description 'fbt::buf_hash:return ' matched 2 probes
CPU     ID                    FUNCTION:NAME
  7     30                        :tick-30s

           value  ------------- Distribution ------------- count
             < 0 |                                         0
               0 |@@                                       20581
               1 |@                                        12578
               2 |@                                        6004
               3 |@@                                       15215
               4 |@                                        4660
               5 |                                         2952
               6 |                                         3678
               7 |@@                                       14091
               8 |                                         2402
               9 |@@                                       20998
              10 |@                                        5805
              11 |@                                        6564
              12 |@@@@                                     35560
              13 |@                                        13021
              14 |                                         4348
              15 |@@                                       17035
              16 |@@                                       15406
              17 |@                                        5512
              18 |@                                        13222
              19 |@                                        5488
              20 |@                                        5404
              21 |@@                                       13583
              22 |@                                        7453
              23 |@                                        4794
              24 |                                         3738
              25 |@@@                                      24918
              26 |@@                                       15566
              27 |@                                        5324
              28 |@                                        12112
              29 |@@                                       13966
              30 |@@                                       16668
              31 |@                                        9904
           >= 32 |                                         0

So that looks reasonably even - every bucket is in use. I think your patch
should be good.

Brendan

-- 
Brendan Gregg, Joyent                      http://dtrace.org/blogs/brendan