Freebsd 8.0 kmem map too small

[Jeremy Chadwick]

On Tue, Jun 08, 2010 at 06:11:46PM +0900, Yoshiaki Kasahara wrote:
> Hello,
> 
> I'd like to add another instance of similar problems.  I recently
> updated my FreeBSD amd64 box with ZFS root and 8GB RAM from 8-STABLE
> (as of Mar 1st) to 8.1-PRERELEASE (as of May 27th).  After that, my
> box started to crash every couple of days due to kmem_map too small.
> 
> Here is a (last week) screenshot of Munin graph about the memory usage
> of the box:
> 
> http://eron.info/munin-memory.png
> 
> In "by month" graph, a white gap at the end of "Week 20" is the update
> period from 8-STABLE to 8.1-PRERELEASE I mentioned above.  Before the
> upgrade, the system was rock solid without any kmem tuning in
> loader.conf (vm.kmem_size was around 2.7GB IIRC).
> 
> After the update, I could see that more wired memory was assigned, and
> then steep drop (crash) occured.
> 
> "by day" graph shows my experiment to bump vm.kmem_size=12G
> (recommended somewhere in this thread) and explicitly limit
> vfs.zfs.arc_max=2G.  I was surprised because the wired memory quickly
> increased over 5GB...
> 
> Then I noticed that the default value of vfs.zfs.zio.use_uma was 1 on
> amd64, so I turned it off and removed other memory tunings (kmem_size
> and arc_max) in loader.conf on Tuesday at 16:00 and rebooted.  It
> seems that the usage of wired memory was stablized and no crash since
> then with the default kmem_size and arc_max.
> 
> Does anyone have any idea about this behavior?
> 
> My wild guess is that ZFS/UMA code is not stable on amd64 too, and it
> should be turned off by default for 8.1-RELEASE maybe....

I realise you're talking about amd64, but I'm not sure how UMA is
getting enabled on i386 to begin with.  It does look like it's enabled
on amd64 by default.

src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c --

#if defined(__amd64__)
static int zio_use_uma = 1;
#else
static int zio_use_uma = 0;
#endif
SYSCTL_DECL(_vfs_zfs);
SYSCTL_NODE(_vfs_zfs, OID_AUTO, zio, CTLFLAG_RW, 0, "ZFS ZIO");
TUNABLE_INT("vfs.zfs.zio.use_uma", &zio_use_uma);
SYSCTL_INT(_vfs_zfs_zio, OID_AUTO, use_uma, CTLFLAG_RDTUN, &zio_use_uma, 0,
    "Use uma(9) for ZIO allocations");

Also note that this tunable does not appear in sysctl -a output; I
believe this is caused by the first parameter to SYSCTL_INT() being
incorrect -- should be _vfs_zfs, not _vfs_zfs_zio.  Regardless of
that, it's still adjustable in loader.conf.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |