vm.kmem_size for stability to avoid kernel panics?

[Jeremy Chadwick]

On Thu, Sep 16, 2010 at 10:39:05PM -0700, Gil Vidals wrote:
> The ZFS Tuning guide says that no tuning may be necessary on servers with 2+
> GB of RAM.
> 
> *"FreeBSD 7.2+ has improved kernel memory allocation strategy and no tuning
> may be necessary on systems with more than 2 GB of RAM."*
> However, is it advisable to put an upper limit for these two parameters to
> ensure stability and avoid kernel panics?
> 
> vm.kmem_size
> vm.kmem_size_max

It depends entirely on what OS version -- and build date -- you're
using, in addition to architecture (i386 vs. amd64).  Please provide
more information about your system.

There were changes to the underlying VM to extend vm.kmem_size_max's
limit, which got committed sometime during the 7.2-STABLE (I think)
cycle.

However, vm.kmem_size still needs to be adjusted on both RELENG_7 and
RELENG_8.  You *do not* (and should not) need to adjust
vm.kmem_size_max.

Another ZFS-centric tunable you should adjust is vfs.zfs.arc_max -- but
again, the functionality of this tunable depends exactly on what OS
version and date of build you're using.

The functionality in vfs.zfs.arc_max was changed from being a "high
watermark" to a hard limit due to people continuing to experience "kmem
map too small" panics.  I think I may have posted to the list long ago
about when this was changed; I don't remember the date off the top of my
head.  I do know that for RELENG_8, vfs.zfs.arc_max is a hard limit.

Each person's hardware, environment, and workload is different, so your
tuning will vary.

This is the ZFS-related tuning bits we use on our amd64 RELENG_7 and
RELENG_8 systems which have 4GB physical RAM installed:

# Increase vm.kmem_size to allow for ZFS ARC to utilise more memory.
vm.kmem_size="4096M"
vfs.zfs.arc_max="3584M"

# Disable ZFS prefetching
# http://southbrain.com/south/2008/04/the-nightmare-comes-slowly-zfs.html
# Increases overall speed of ZFS, but when disk flushing/writes occur,
# system is less responsive (due to extreme disk I/O).
# NOTE: 8.0-RC1 disables this by default on systems <= 4GB RAM anyway
# NOTE: System has 8GB of RAM, so prefetch would be enabled by default.
vfs.zfs.prefetch_disable="1"

# Disable UMA (uma(9)) for ZFS; amd64 was moved to exclusively use UMA
# on 2010/05/24.
# http://lists.freebsd.org/pipermail/freebsd-stable/2010-June/057162.html
vfs.zfs.zio.use_uma="0"

# Decrease ZFS txg timeout value from 30 (default) to 5 seconds.  This
# should increase throughput and decrease the "bursty" stalls that
# happen during immense I/O with ZFS.
# http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007343.html
# http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007355.html
vfs.zfs.txg.timeout="5"

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |