kmem_map too small with ZFS and 8.2-RELEASE

[Jeremy Chadwick]

On Mon, Mar 07, 2011 at 10:49:26AM +0100, Martin Matuska wrote:
> In 8-STABLE (amd64), starting with SVN revision 214620
> vm.kmem_size_scale defaults to 1.
> 
> This means in 8.2-RELEASE, vm.kmem_size is automatically set to the
> amout of your system RAM,
> so vm.kmem_size="16G" is automatically set on a system with 16GB RAM.
>
> D??a 04.03.2011 11:05, Jeremy Chadwick wrote / nap??sal(a):
> > On Fri, Mar 04, 2011 at 10:48:53AM +0100, Micka??l Can??vet wrote:
> >>> I'd use vm.kmem_size="32G" (i.e. twice your RAM) and that's it.
> >> Should I also increase vfs.zfs.arc_max ?
> > You should adjust vm.kmem_size, but not vm.kmem_size_max.
> >
> > You can adjust vfs.zfs.arc_max to basically ensure system stability.
> > This thread is acting as evidence that there are probably edge cases
> > where the kmem too small panic can still happen despite the limited ARC
> > maximum defaults.
> >
> > For a 16GB system, I'd probably use these settings:
> >
> > vm.kmem_size="16384M"
> > vfs.zfs.arc_max="13312M"
> >
> > I would also use these two settings:
> >
> > # Disable ZFS prefetching
> > # http://southbrain.com/south/2008/04/the-nightmare-comes-slowly-zfs.html
> > # Increases overall speed of ZFS, but when disk flushing/writes occur,
> > # system is less responsive (due to extreme disk I/O).
> > # NOTE: Systems with 8GB of RAM or more have prefetch enabled by
> > # default.
> > vfs.zfs.prefetch_disable="1"
> >
> > # Decrease ZFS txg timeout value from 30 (default) to 5 seconds.  This
> > # should increase throughput and decrease the "bursty" stalls that
> > # happen during immense I/O with ZFS.
> > # http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007343.html
> > # http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007355.html
> > vfs.zfs.txg.timeout="5"
> >
> > The advice in the Wiki is outdated, especially for 8.2-RELEASE.  Best
> > not to follow it as of this writing.
> >
> >> Do you have any idea why the kernel panicked at only 8GB allocated ?
> > I do not.  A kernel developer will have to comment on that.
> >
> > Please attempt to reproduce the problem.  If you can reproduce it
> > reliably, this will greatly help kernel developers tracking down the
> > source of the problem.

Thanks -- you're absolutely right.  I had forgotten all about
vm.kmem_size_scale.  :-)

So yeah, with 8.2-RELEASE onward (rather than get into individual SVN
revisions I'm using 8.2-RELEASE as "a point in time"), vm.kmem_size will
default to the amount of usable memory (usually slightly less than
hw.physmem).  Validation:

$ sysctl hw.realmem hw.physmem hw.usermem
hw.realmem: 9395240960
hw.physmem: 8579981312
hw.usermem: 1086521344
$ sysctl vm.kmem_size
vm.kmem_size: 8303894528

As such, one only needs to tune vfs.zfs.arc_max if desired.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |