ZFS: Disabling ARC?

[Paul Kraus]

On Jul 31, 2015, at 20:37, Quartz <quartz at sneakertech.com> wrote:

> Can someone help clear up a few ZFS basics for me?
> 
> A few recent threads about ARC issues and memory-induced panics have made me realize I'm not 100% sure I understand ARC as well as I thought I did.

<snip>

> If I understand ARC correctly this would be a worst case scenario, right? Besides hogging ram, would ARC cause any problems here? Would disabling ARC and devoting the ram to other things be a wise idea? Is disabling ARC ever a wise idea?

The ZFS ARC is both powerful and often misunderstood. In order to do anything intelligently with the ARC (and it’s tunings) you need to first know what it is doing… get a copy of arcstat.pl from here https://github.com/mharsch/arcstat/blob/master/arcstat.pl … I prefer it to the zfs-stats port, but any tool that shows us the realtime ARC usage is what you need.

The ARC is _supposed_ to use whatever RAM is otherwise unused _and_ if other processes need RAM, the ARC is supposed to give it back. I know there have been a number of issues with the ARC not freeing RAM fast enough, but I have not seen that issue under 10.1 (I have seen it under 9.x). On my main 9.x VM host I leave an 'arcstat.pl 60' running all the time so I can quickly see if it is freeing as it should.

Also note that the ARC does not cache files but blocks (if, as Alex mentioned) it is cacheing data at all (or just metadata).

If you are really worried about the ARC hogging RAM, then set a cap. The kernel tunables here are:

[ppk at FreeBSD2 ~]$ sysctl -a | grep zfs | grep arc
vfs.zfs.arc_max: 12884901888
vfs.zfs.arc_min: 1919138304
vfs.zfs.arc_meta_used: 3221144040
vfs.zfs.arc_meta_limit: 3221225472
vfs.zfs.l2arc_write_max: 8388608
vfs.zfs.l2arc_write_boost: 8388608
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_noprefetch: 1
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_norw: 1
… followed by _lots_ of arcstats, which tell you what the ARC is doing. 

The first one is what you really want to tune if you are RAM limited. It sets a hard cap on how much RAM the system will use for the entire ARC. On my home server I have the following tunings in /boot/loader.conf

[ppk at FreeBSD2 /boot]$ cat loader.conf 
zfs_load="YES"
# set ARC max to 12 GB with 16 GB of RAM
vfs.zfs.arc_max="12884901888"

So I set the ARC maximum to 12 GB to leave me 4 GB for OS and applications.

I have production servers with between 24 and 80 GB of RAM and do not generally limit the ARC but let it “float” and give and take as RAM is available.

Please note that I have never seen a _panic_ due to ARC RAM issues, I have had systems starved for RAM for periods and processes (VMs) get very angry, but the system as a whole usually recovers. I then restart the processes that got angry. 

--
Paul Kraus
paul at kraus-haus.org