Freebsd 8.0 kmem map too small

[Paul Schenkeveld]

On Fri, Jun 11, 2010 at 04:12:50PM +0200, Florian Smeets wrote:
> On 6/8/10 11:11 AM, Yoshiaki Kasahara wrote:
> > Hello,
> > 
> > I'd like to add another instance of similar problems.  I recently
> > updated my FreeBSD amd64 box with ZFS root and 8GB RAM from 8-STABLE
> > (as of Mar 1st) to 8.1-PRERELEASE (as of May 27th).  After that, my
> > box started to crash every couple of days due to kmem_map too small.
> > 
> > Here is a (last week) screenshot of Munin graph about the memory usage
> > of the box:
> > 
> > http://eron.info/munin-memory.png
> > 
> > In "by month" graph, a white gap at the end of "Week 20" is the update
> > period from 8-STABLE to 8.1-PRERELEASE I mentioned above.  Before the
> > upgrade, the system was rock solid without any kmem tuning in
> > loader.conf (vm.kmem_size was around 2.7GB IIRC).
> > 
> > After the update, I could see that more wired memory was assigned, and
> > then steep drop (crash) occured.
> > 
> > "by day" graph shows my experiment to bump vm.kmem_size=12G
> > (recommended somewhere in this thread) and explicitly limit
> > vfs.zfs.arc_max=2G.  I was surprised because the wired memory quickly
> > increased over 5GB...
> > 
> > Then I noticed that the default value of vfs.zfs.zio.use_uma was 1 on
> > amd64, so I turned it off and removed other memory tunings (kmem_size
> > and arc_max) in loader.conf on Tuesday at 16:00 and rebooted.  It
> > seems that the usage of wired memory was stablized and no crash since
> > then with the default kmem_size and arc_max.
> > 
> > Does anyone have any idea about this behavior?
> > 
> > My wild guess is that ZFS/UMA code is not stable on amd64 too, and it
> > should be turned off by default for 8.1-RELEASE maybe....
> > 
> 
> I can confirm this. I just upgraded a system from late January to
> yesterdays 8-STABLE. I had to set vfs.zfs.zio.use_uma="0", to get the
> system survive buildworld again.
> 
> This is amd64 with only 1GB, (i know this is considered not enough for
> ZFS) with the kernel from January it was stable without any tuning.
> After disabling ZFSs UMA usage, it seems to be working fine again.
> 
> Just another data point.

+1 for me, several of my servers crash *every night* on a script that
does csup from cvs.freebsd.org (cvs mode if that matters) followed by
cvs up -dP in some source trees and one ports tree.

Hardware has 12 GB RAM, dual nehalem processors; kernel is amd64, no
sysctl tuning related to ZFS.  With kernels before april 18 everything
is rock solid here, one production server has been up since april 17.

Can do some limited testing on one of these servers on mondays if
needed.

> Cheers,
> Florian

Regards,

Paul Schenkeveld