zfs arc - just take it all and be good to me

[Artem Belevich]

> The bottom line of my original mail was:
> - Hey, I set the arc to just take all my RAM
> - It grows on reads and writes
> - It shrinks when programs need the memory it has.
> - It behaves well in a scenario of combined read/write/other programs.
> - This is good, really good, what is dangerous about my settings that
>  not everybody just sets the arc to nearly all their physical memory on
>  FreeBSD?

The issue is that ARC will give up memory even if there's plenty of it
available and sitting in inactive/cache queues. That's particularly
nasty in case your system uses some other filesystem besides ZFS. For
example try tarring up few gigabytes worth of data from UFS filesystem
and see how far your ARC size shrinks. It could be mitigated by
setting minimum ARC size to be large enough so ZFS performance does
not degrade. The downside is that ARC will not give up memory below
its minimum no matter what, so if your APP really needs it, it would
have to go to swap.

There's a hack floating around that attempts to force kernel into
freeing up memory from inactive/cache lists before draining ARC. It
does help a bit with this issue, but it's still a hack.

--Artem