ZFS ARC being limited below what is defined in /boot/loader.conf

[Jeremy Chadwick]

On Fri, Feb 12, 2010 at 01:10:51PM -0800, Artem Belevich wrote:
> vm.kmem_size_max/vm.kmem_size_min define the range vm.kmem_size can be set to.
> vm_kmem_size specifies the actual kmem size.
> 
> ARC size in turn limited by vm.kmem_size.
> 
> If you want to bump ARC size, you do need to bump vm.kmem_size.
> 
> --Artem
> 
> On Fri, Feb 12, 2010 at 11:36 AM, Steve Polyack <korvus at comcast.net> wrote:
> > On 02/12/10 13:47, Artem Belevich wrote:
> >>
> >> On Fri, Feb 12, 2010 at 10:31 AM, Steve Polyack<korvus at comcast.net>
> >>  wrote:
> >>
> >>
> >>>
> >>> Has anyone had an issue with the ZFS ARC max being limited below what has
> >>> been defined in /boot/loader.conf?  I just upgraded the RAM in a
> >>> ZFS-equipped system and attempted to devote 4GB to the ARC cache by
> >>> placing
> >>> the following in loader.conf:
> >>>  vfs.zfs.arc_max="4096M"
> >>>
> >>> However, after rebooting, querying the sysctl gives me this:
> >>> $ sysctl vfs.zfs.arc_max
> >>> vfs.zfs.arc_max: 1726489600
> >>>
> >>> or about 1.7GB, an odd number that I can't find any references to.  For
> >>> reference, I'm running 8-STABLE (as of Jan 19th) on an amd64 system with
> >>> 8GB
> >>> of RAM.  The system was previously very stable with 4GB of RAM and a
> >>> 512MB
> >>> arc_max.  I have not modified vm.kmem_size_max (defaults to ~330GB on
> >>> amd64)
> >>> or any other ZFS tunables.  I'd also like to avoid syncing up to the
> >>> current
> >>> 8-STABLE if at all possible.
> >>>
> >>> Thanks,
> >>> Steve Polyack
> >>>
> >>> _______________________________________________
> >>> freebsd-stable at freebsd.org mailing list
> >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
> >>>
> >>>
> >>
> >> Check your vm.kmem_size. Default setting is way too low. Set it to at
> >> least double of desired arc size.
> >>
> >> --Artem
> >
> > I mentioned it briefly, but vm.kmem_size_max was left at the default for
> > amd64.  At 330GB it is way above and beyond what will ever be allocated to
> > ARC:
> > $ sysctl vm.kmem_size_max
> > vm.kmem_size_max: 329853485875

I'm concerned about adjusting vm.kmem_size, since vm.kmem_size_max is
supposed to be auto-adjusting as of this point in time.

How does adjusting vm.kmem_size affect things like kern.maxdsiz,
kern.dfldsiz, and kern.maxssiz?  These tunings are required for things
like userland apps which require a large amount of memory (read:
mysqld).

The amount of VM-related tunables requiring is getting out of hand.  We
need documentation of some sort, outlining how all these things fit
together as to avoid potential kernel panics or memory exhaustion
issues; for example, people with 8GB of RAM installed who utilise both
ZFS on the server in addition to a memory-hungry mysqld.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |