kmem_map too small with ZFS and 8.2-RELEASE

[Jeremy Chadwick]

On Fri, Mar 04, 2011 at 10:48:53AM +0100, Mickaël Canévet wrote:
> > I'd use vm.kmem_size="32G" (i.e. twice your RAM) and that's it.
> 
> Should I also increase vfs.zfs.arc_max ?

You should adjust vm.kmem_size, but not vm.kmem_size_max.

You can adjust vfs.zfs.arc_max to basically ensure system stability.
This thread is acting as evidence that there are probably edge cases
where the kmem too small panic can still happen despite the limited ARC
maximum defaults.

For a 16GB system, I'd probably use these settings:

vm.kmem_size="16384M"
vfs.zfs.arc_max="13312M"

I would also use these two settings:

# Disable ZFS prefetching
# http://southbrain.com/south/2008/04/the-nightmare-comes-slowly-zfs.html
# Increases overall speed of ZFS, but when disk flushing/writes occur,
# system is less responsive (due to extreme disk I/O).
# NOTE: Systems with 8GB of RAM or more have prefetch enabled by
# default.
vfs.zfs.prefetch_disable="1"

# Decrease ZFS txg timeout value from 30 (default) to 5 seconds.  This
# should increase throughput and decrease the "bursty" stalls that
# happen during immense I/O with ZFS.
# http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007343.html
# http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007355.html
vfs.zfs.txg.timeout="5"

The advice in the Wiki is outdated, especially for 8.2-RELEASE.  Best
not to follow it as of this writing.

> Do you have any idea why the kernel panicked at only 8GB allocated ?

I do not.  A kernel developer will have to comment on that.

Please attempt to reproduce the problem.  If you can reproduce it
reliably, this will greatly help kernel developers tracking down the
source of the problem.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |