vm.kmem_size_scale recommendation for ZFS

Jeremy Chadwick freebsd at jdc.parodius.com
Tue Oct 4 04:56:11 UTC 2011


On Tue, Oct 04, 2011 at 03:10:40PM +1100, Aristedes Maniatis wrote:
> Having just suffered from a FreeBSD 8.2 machine hit with the  "kmem_map too small" error [1], what is the current recommendation for a machine running RELEASE-8.2?
> 
> A. vm.kmem_size_scale=1 (this is the new default in 8-STABLE but 8.2-RELEASE had it set to 3)
> B. vm.kmem_size_scale=0.66 (this is aligned with Pawel's recommendation of setting vm.kmem_size to 150% of your RAM. [2]
> C. Leaving vm.kmem_size_scale alone and manually setting vm.kmem_size directly as Pawel is doing
>
> [1] http://freebsd.1045724.n5.nabble.com/kmem-map-too-small-with-ZFS-and-8-2-RELEASE-td4029979.html
> [2] http://blogs.freebsdish.org/pjd/2010/08/06/from-sysinstall-to-zfs-only-configuration

Even if you were to mess about with vm.kmem_size_scale (which IMO should
be set to 1 -- it defaults to that in 8.2-STABLE as well as other
upcoming releases), you will still experience problems with kmem map
exhaustion **even if** you limit the ARC.

There are fixes in 8.2-STABLE that address this problem (there were
multiple fixes put in place).  So if you're already experiencing kmem
map exhaustion, I strongly recommend you upgrade to 8.2-STABLE and then
adjust one single tunable:

vfs.zfs.arc_max

You didn't disclose how much RAM your machine has, nor what other
daemons are running on it, so it's very hard to give you an estimate.

I tend to tell people to set vfs.zfs.arc_max to about 60% of their
memory.  E.g. if the machine has 8GB RAM, set vfs.zfs.arc_max="5120M".
We use this on all of our servers with 8GB, including ones that run
mysqld with some tunings.  Example machine in question:

  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
60030 mysql      16  76    0   785M   294M sigwai  0  15:38  0.00% [mysqld]

And relevant bits from /boot/loader.conf:

kern.maxdsiz="2560M"
kern.dfldsiz="2560M"
kern.maxssiz="256M"
vfs.zfs.arc_max="5120M"

You need to keep in mind that ZFS can still use more than what you limit
the ARC to.  The mailing list has explanations regarding this; I believe
it has to do with fragmentation or something like that.  So don't go
thinking you're being smart by setting it to something like "7500M";
there is a very good chance you will still experience the same problem.
So my advice is to "start small" and work your way up after multiple
weeks (not days!) of utilising the filesystems on ZFS, to really stress
the ARC.

Also be aware there may still be problems with applications that use
sendfile(2) -- known programs which use this are ftpd (not adjustable),
Apache (you can disable it), nginx (I think you can disable it), and
Samba (you can disable it).  There are some sendfile(2) fixes were which
committed, but I'm not 100% sure if all situations have been accounted
for.

P.S. -- I am assuming you're using amd64.  If you're using i386 this
whole situation becomes much, much more complex.

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                   Mountain View, CA, US |
| Making life hard for others since 1977.               PGP 4BD6C0CB |



More information about the freebsd-fs mailing list