ZFS: Disabling ARC?

[Matt Churchyard]

>On Fri, 31 Jul 2015 20:37:11 -0400
>Quartz <quartz at sneakertech.com> wrote:

> Can someone help clear up a few ZFS basics for me?
> 
> A few recent threads about ARC issues and memory-induced panics have 
> made me realize I'm not 100% sure I understand ARC as well as I 
> thought I did.
> 
> Say you have a ZFS file server that houses very large single files 
> which are very infrequently accessed. For the sake of argument, let's 
> say you're using ZFS on a home server for your family, and it holds 
> exclusively a whole bunch of multi-gig bluray rips or whatever 
> (nothing else). When someone wants to watch something, they copy the 
> file to their desktop and watch it there. Although the family will 
> watch several videos each day, any given file will only be accessed 
> maybe once every couple months. (I know streaming would make more 
> sense in real life, and that this example is kinda silly in general, 
> but ignore that for now).

>No matter if you stream or copy, it's the same operation, read once in a while.

> If I understand ARC correctly this would be a worst case scenario, 
> right? Besides hogging ram, would ARC cause any problems here? Would 
> disabling ARC and devoting the ram to other things be a wise idea? Is 
> disabling ARC ever a wise idea?

>You can tune how the ARC is used:

># zfs get all space/export/Movies | grep cache
>space/export/Movies  primarycache          metadata               local
>space/export/Movies  secondarycache        none                   local

>"primarycache" is the ARC in RAM, "secondarycache" is a cache device / L2ARC (SSD).

>"metadata" is directory listing, file sizes, access permissions and the like.

>So the above example means that metadata is allowed to go to the ARC in RAM, and nothing of the real data in this dataset shall be cached >anywhere at all (neither in a cache device nor in RAM).

>Bye,
>Alexander.

>--
>http://www.Leidinger.net Alexander at Leidinger.net: PGP 0xC773696B3BAC17DC
>http://www.FreeBSD.org    netchild at FreeBSD.org  : PGP 0xC773696B3BAC17DC

I don't know if it's changed, but even changing ARC to metadata only used to cause massive performance issues when reading large files in small chunks. Reading a 128k ZFS record in 4k chunks would cause ZFS to read the same 128k record from disk 32 times.
There's a forum thread about it here - https://forums.freebsd.org/threads/zfs-primarycache-all-versus-metadata.45555/

Generally I've found ARC to be one of the most important parts of ZFS. There's no case I know of where it will actually adversely affect performance (not that I've really looked). Its only downside is that it doesn't seem to manage memory as well as it should, and so a lot of people (including me) have resorted to limiting it.

While it seems easy to "tune" using the primarycache option, it's rarely a good idea except for very specific situations.

If ARC isn't getting many hits (I use sysutils/zfs-stats to view stats), and you really don't want it using up all your memory, just reduce arc_max.

Matt