FreeBSD 8.1-Prerelease Panic amd64 w/ZFS..

[Jeremy Chadwick]

On Tue, Jun 01, 2010 at 11:54:55AM -0700, Darren Pilgrim wrote:
> Jeremy Chadwick wrote:
> >To the OP: you will need to increase vm.kmem_size in /boot/loader.conf
> >and reboot the system.  "What value do I pick?"  With 4GB, I would
> >recommend you use these two variables:
> >
> >vm.kmem_size="2048M"
> >vfs.zfs.arc_max="1536M"
> >
> >This will increase the available kmem, and also limit the ARC size
> >explicitly to nothing larger than 1.5GB.  This should stabilise your
> >system.
> 
> What are the general guidelines for sizing these given the amount of
> physical RAM?  For example, I have a machine with 6 GB, so I have:
> 
> vm.kmem_size="3072M"
> vfs.zfs.arc_max="2048M"

There really aren't any, as far as I know, since it depends on each
system, it's role, and what architecture type (i386 vs. amd64).  I'm
focusing solely on amd64.

Method I tend to follow:

vm.kmem_size    = 1/2 of available RAM (think physical/installed)
vfs.zfs.arc_max = vm.kmem_size minus 512MB

I believe there were some reported cases of panics on systems with
vm.kmem_size == vfs.zfs.arc_max, but I could be remembering incorrectly.
There's too many posts/people to remember who said what and when.  I
think there's a kstat.zfs.misc.arcstats sysctl counter which tracks
the maximum ARC ever used, but could be mistaken here too.

One of our 8GB systems that uses ZFS also runs MySQL, which requires me
to use these values in addition to the above.  Note that, AFAIK, the
below values *do not* affect ZFS directly; I'm pointing them out as
necessities, else a "heavy" mysqld crashes and complains about there
not being enough memory (directly correlating to maxdsiz).

kern.maxdsiz="1536M"
kern.dfldsiz="1536M"
kern.maxssiz="256M"

Rather than try to pick values that "all add up to 8GB" (don't do this),
I leave a lot of extra space for other things (kernel bits, network
buffers, blah blah) that I don't want to think about.  I try to avoid
being OCD with tuning unless I have reason to be.  All I really care
about is providing enough memory for userland daemons and ZFS to run
reliably/not panic the box.

I'd welcome the community and developers to chime in here, especially
folks familiar with the VM and how all of these tunables interact with
one another.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |