zfs arc - just take it all and be good to me

[Marco van Tol]

On Tue, Aug 10, 2010 at 06:49:19PM -0700, Jeremy Chadwick wrote:
> On Tue, Aug 10, 2010 at 09:44:18PM +0000, Marco van Tol wrote:
> > [...]
> > 
> > All in all this looks like a close attempt at zfs memory being auto
> > tuned while using maximum amount of memory.  The only problem is, nobody
> > else is doing it like this so its very likely that this is not the smart
> > thing to do.
> 
> I'm not sure what "nobody else is doing it like this" means, but Solaris
> 10 behaves exactly as you describe -- the ARC takes up as much memory as
> it can.  When an application or other piece of the kernel wants memory
> which the ARC can free up, it releases memory.  There's no tuning
> required on Solaris; it "just works".

Hm by "nobody else is doing it like this" I meant on FreeBSD.  I also
meant to say that what you will typically see people advising/doing is
setting arc_max to a value much less then their total physical memory.
What I have done instead is set arc_max to nearly all my physical
memory (1792G out of a total of 2GB).

In combination with that I set kmem_size to 1.5x physical memory to
account for kernel fragmentation.  Here is a post that touches that
subject:
http://lists.freebsd.org/pipermail/freebsd-fs/2010-January/007614.html

What I am glad to see happening so far, but I haven't given the system
any production levels of stress, is that the arc happely takes up nearly
all my memory, and just surrenders it to programs that happen to run,
taking it back after they finish.  Very nice.
(I'll feed it bonny++ and see what happens)

My mail wasn't intended to make any complaints, much the contrairy.  I
am a happy camper ever since I changed the test system to those
settings.  I think/hope that was clear. :)

> At my day job (where we use Solaris 10), when we introduced ZFS into the
> picture, many of our memory usage monitors began firing indicating lack
> of free memory due to ARC usage.  We had to add some code to our
> check_system_memory monitor which called kstat to examine the
> zfs:0:arcstats:size and zfs:0:arcstats:c_min properties and then add
> that to the overall amount of memory available.  Code in question:
> 
> my $zfsarcstatssize = "zfs:0:arcstats:size";
> my $zfsarcstatsmin = "zfs:0:arcstats:c_min";
> my $zfsarcsize;
> my $zfsarcmin = 0;
> if ($OS eq 'solaris' and -x $kstat) {
>     my @dump = `$kstat -p $zfsarcstatssize $zfsarcstatsmin`;
>     unless ($?) {
>         foreach my $dump (@dump) {
>             chomp $dump;
>             my ($label, $size) = split(/\t/, $dump);
>             if ($label eq $zfsarcstatssize) {
>                 $zfsarcsize = sprintf("%.2f", $size/1024/1024);
>             }
>             if ($label eq $zfsarcstatsmin) {
>                 $zfsarcmin = sprintf("%.2f", $size/1024/1024);
>             }
>         }
>         $real_available += $zfsarcsize - $zfsarcmin if (defined $zfsarcsize and $zfsarcsize > $zfsarcmin);
>     }
> }

Its nice when you can rely on something to act equally to actual free
memory, yet provide great speedups at times it can use that memory.

> I believe OpenSolaris behaves the same as Solaris 10 in this regard.
> Their VM model differs from that of FreeBSD.
> 
> I've talked about this in the past (see paragraph starting with "Fast
> forward to today").  I can dig up John's original response (talking
> about how the VM needs to be improved in FreeBSD to accomplish the
> equivalent of what Solaris does) if need be.
> 
> http://lists.freebsd.org/pipermail/freebsd-fs/2010-May/008598.html

I had seen that post before indeed.  Thank you.

The bottom line of my original mail was:
- Hey, I set the arc to just take all my RAM
- It grows on reads and writes
- It shrinks when programs need the memory it has.
- It behaves well in a scenario of combined read/write/other programs.
- This is good, really good, what is dangerous about my settings that
  not everybody just sets the arc to nearly all their physical memory on
  FreeBSD?

That's all, :)

Thanks,

Marco

-- 
A male gynecologist is like an auto mechanic who never owned a car.
- Carrie Snow