zfs l2arc warmup

Fri Mar 28 09:23:37 UTC 2014

On 28 March 2014 01:59,  <kpneal at pobox.com> wrote:
> On Thu, Mar 27, 2014 at 11:10:48AM +0100, Joar Jegleim wrote:
>> But it's really not a problem for me how long it takes to warm up the
>> l2arc, if it takes a week that's ok. After all I don't plan on
>> reboot'ing this setup very often + I have 2 servers so I have the
>> option to let the server warmup until i hook it into production again
>> after maintenance / patch upgrade and so on .
>>
>> I'm just curious about wether or not the l2arc warmup itself, or if I
>> would have to do that manual rsync to force l2arc warmup.
>
> Have you measured the difference in performance between a cold L2ARC and
> a warm one? Even better, have you measured the performance with a cold
> L2ARC to see if it meets your performance needs?
No I haven't.
I actually started using those 2 ssd's for l2arc the day before I sent
out this mail to the list .
I haven't done this the 'right' way by producing some numbers for
measurement, but I do know that the way this application work today is
that it will pull random jpegs from this dataset of about 1.6TB,
consisting of lots of millions of files ( more than 20 million). And
that today this pool is served from 20 SATA 7.2K disks which would be
the slowest solution for random read access.
Based on the huge performance gain by using ssd's simply by looking at
the spec., but also by looking at other peoples graphs from the net (
people who have done this more thorough than me) I'm pretty confident
to say that if at any time when the application request a jpeg if it
was served from either ram or ssd it would be a substantial
performance gain compared from serving it from the 7.2k array of
disks.

>
> If you really do need a huge L2ARC holding most of your data have you
> considered that maybe you are taking the wrong approach to getting
> performance? Consider load balancing across multiple servers, or having
> your application itself spread the loads of pictures across multiple
> servers.
yes I have  :p but again that would mean I'd have to rewrite the
application, or I would have to have several servers mirrored. There
are problems with having several servers mirrored related to the
application, I'll skip those details here, but I have thought about
what if I served those jpegs from say 4 backend servers, I really
don't think it would help compared to serving stuff from ssd's, or I
would at least have to have 20 disks pr. server for it to be any
performance gain... Bu t I'd still have latency and all the
disadvantages from having 7.2k disks.

The next release of the application actually has taken this into
account and I will in the future be able to spread this over 4
servers.
For the future I might spread this stuff over more backends.
At the moment the cheapest and easiest would be to simply by 2 more
480GB ssd's, put them in the server and make sure as much as possible
of the dataset resides in l2arc.

>
> If a super-huge L2ARC is really needed for the traffic _today_, what about
> when you have more traffic in 3-6-12 months? What about if you increase
> the number of pictures you are randomly choosing from? If your server is
> at the limit of its performance today then pretty soon you will outgrow
> it. Then what?
The server is actually far from any limit, in fact it has so 'little'
to do I've been a bit put off to figure out why our frontpage won't be
more snappy.
And these things will probably be taken care of, again, in the next
release of the application which will give me control of 'todays'
frontpage mosaic pictures where I can either make sure frontpage jpegs
stay in arc, or I'll simply serve frontpage jpegs from varnish .

>
> What happens if your production server fails and your backup server has
> a cold L2ARC? Then what?
performance would drop, but nothing really serious + I got 2 of them,
and my plan is to make sure the l2arc for the second server is warm.

>
> Having more and more parts in a server also means you have more opportunities
> for a failure, and that means a higher chance of something bringing down
> the entire server. What if one of the SSD in your huge L2ARC fails in a
> way that locks the bus? This is especially important since you indicated
> you are using cheaper SSD for the L2ARC. Fewer parts -> more robust server.
Good point. Again, I have a failover server and a proxy with health
check in front, and actually I have a third 'fall-back' server too for
worst case scenarios.

>
> On the ZIL: the ZIL holds data on synchronous writes. That's it. This is
> usually a fraction of the writes being done except in some circumstances.
> Have you measured to see if, or do you otherwise know for sure, that you
> really do need a ZIL? I suggest not adding a ZIL unless you are certain
> you need it.
Yes, I only recently realized that too, and I'm really not sure if a
zil is required.
Some small portion of files (som hundre MB's) are served over nfs from
the same server, if I understand it right a zil will help for nfs
stuff (?) , but I'm not sure if it's any gain of having a zil today.
On the other hand, a zil doesn't have to be big, I can simply buy a
128GB ssd which are cheap today .

>
> Oh, and when I need to pull files into memory I usually use something
> like 'find . -type f -exec cat {} \; >/dev/null'. Well, actually, I
> know I have no spaces or special characters in filenames so I really
> do 'find . -type f -print | xargs cat > /dev/null'. This method is
> probably best if you use '-print0' instead plus the correct argument to
> xargs.
Thanks, this really makes sense and I reckon it would be faster than
rsync from an other server.

>
> --
> Kevin P. Neal                                http://www.pobox.com/~kpn/
>
> "Nonbelievers found it difficult to defend their position in \
>     the presense of a working computer." -- a DEC Jensen paper

-- 
----------------------
Joar Jegleim
Homepage: http://cosmicb.no
Linkedin: http://no.linkedin.com/in/joarjegleim
fb: http://www.facebook.com/joar.jegleim
AKA: CosmicB @Freenode

----------------------