Panic in ZFS layer on 8.1-STABLE

[Jeremy Chadwick]

On Wed, Dec 15, 2010 at 11:55:40PM +0100, Jeremie Le Hen wrote:
> On Wed, Dec 15, 2010 at 07:28:57PM +0200, Andriy Gapon wrote:
> > on 15/12/2010 10:28 Jeremie Le Hen said the following:
> > > Hi,
> > > 
> > > [ Please Cc: me when replying, as I'm not subscribed to -stable at . ]
> > > 
> > > My filer at home runs FreeBSD.  A single data RAID-1 zpool with 10~15
> > > datasets, two of them using compression.  Over the night, I got the
> > > following panic:
> > 
> > Thanks for the stack trace!
> > But where is the promised panic message? :)
> 
> Eheh, sorry for this :-).
> 
> > I suspect that you ran out of kernel address space.
> > You'd probably have to tune your system and/or add more memory.
> > Please research this topic via mailing lists archives.
> 
> You guessed right!
> 
>   panic: kmem_malloc(114688): kmem_map too small: 775847936 total allocated

Adjust the following two tunables (loader.conf, not sysctl) in
combination:

vm.kmem_size
vfs.zfs.arc_max

I strongly recommend setting vfs.zfs.arc_max to about half of your
system memory in your case (so, vfs.zfs.arc_max="512M").  For example,
on a 4GB system, I use:

vm.kmem_size="4096M"
vfs.zfs.arc_max="3072M"

Make sure vm.kmem_size is larger than arc_max.

Other people will advocate increasing your physical RAM, which I agree
with, but the problem isn't technically the amount of RAM you have, it's
that (paraphrasing to keep it simple) the kernel ran out of usable
memory to use, even though the permissible size (vm.kmem_size_max --
note the difference in the tunable) is much larger.

So, what I'm saying: you can use the above two tunables I mentioned to
effectively ensure you don't encounter this problem.  If after tuning
you DO encounter this *exact same problem* (kmem_map too small), please
report back, as there may be another bug with ARC limiting that needs to
be looked at.  (Every situation is unique/different)

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |