Fileserver panic - FreeBSD 8.1-stable and zfs

[Jeremy Chadwick]

On Fri, Jun 03, 2011 at 12:39:40AM +0200, Torfinn Ingolfsen wrote:
> On Thu, 02 Jun 2011 12:50:26 -0700
> Jeremy Chadwick <freebsd at jdc.parodius.com> wrote:
> > 
> > This is a well-known thing with ZFS on FreeBSD.  Because you're running
> > 8.1-STABLE, this makes figuring out all the tunables and so on a lot
> > more difficult than if you were running 8.2-STABLE.
> 
> FWIW, the machine has been quite stable for me for a long time.

I've tried to explain this to people in the past (not sure if I did to
you or not), but nobody ever listens.  :-)  You can't tune ZFS on
FreeBSD, let it run for 24 hours -- or even 2 months -- and say "it's
stable!"  The exact same problem (specifically the kmem exhaustion
issue) can appear many months later.  It all depends on load, I/O
history, and many other conditionals.  Case in point.

There are certain conditions where ZFS will take up more memory than
just what's in the ARC.  E.g. if you limit the ARC to 2GB and use top,
on occasion you'll see ZFS's ARC usage (usually "Wired", but that also
contains non-ZFS stuff; good luck segregating it!) at maybe 2.5GB.
This is Normal(tm) and Expected(tm).  So when tuning the ARC, you need
to take into consideration that situation.  This is why I tell people to
avoid setting their ARC size to something "too close" to vm.kmem_size.
You really do have to give ZFS "some room" memory-wise.

This advice is with 8.2-STABLE in mind.  The same advice applies to
8.1-STABLE, but different tunings are required.

> > Please provide:
> > 
> > 1) Contents of /boot/loader.conf
> 
> root at kg-f2# more /boot/loader.conf
> zfs_load="YES"
> vfs.root.mountfrom="zfs:zroot"
> siis_load="YES"
> amdtemp_load="YES"
> # testing without MSI
> hw.pci.enable_msix="0"
> hw.pci.enable_msi="0"

Now I'm talking about 8.1-STABLE, but I'm going purely off of memory
here, and like I said in a previous Email at this point in time I've
focused on remembering tunings for 8.2-STABLE, so please keep that in
mind.

The above confirms you didn't tune vm.kmem_size and vm.kmem_size_max,
nor did you try to limit the ARC (vfs.zfs.arc_max).  Therefore I'm not
surprised you're seeing said panic.  Keep reading for tuning advice.

> > 2) Output from: sysctl hw.physmem hw.usermem hw.realmem (your hardware
> >    page says 4GB, but I can't be bothered to sift through multi-pages
> >    of wiki documents and links to find the answers)
> 
> root at kg-f2# sysctl hw.physmem hw.usermem hw.realmem
> hw.physmem: 4141920256
> hw.usermem: 3721527296
> hw.realmem: 4966055936

Thanks -- this helps.

> > 3) Output from: sysctl vfs.zfs.zio.use_uma
> 
> root at kg-f2# sysctl vfs.zfs.zio.use_uma
> vfs.zfs.zio.use_uma: 0

Cool, it's already off on your system, which is good.

> > The "trick" is to tune /boot/loader.conf until you can gain stability.
> 
> Well, the server has been reasonably stable for me for about a year now
> (I had to replace a failing hard drive, but I count that as "wear" not "instability").

I'm amazed it's run for a year.  The I/O on this system must be very
light, maybe kernel memory pressure hasn't been very high (until now;
e.g. maybe other applications on the machine haven't grown in size) or
you must have been **extremely** lucky.  It's impossible to explain why
it took a year, but it's pretty obvious why the problem happened.

> > Again, because you're running 8.1-STABLE, the tuning parameters here
> > will behave different than on 8.2-STABLE.  We can go over those in a
> > follow-up thread.
> 
> I have no trouble with upgrading the server to 8.2-stable, if now is a good time to do it.
> (I haven't watched closely for any zfs related problems on the mailing list lately.)
> 
> > I've gotten to the point where I literally cannot remember all of the
> > different situations/conditions/tunings for each FreeBSD kernel build,
> > release, date, type, etc., so I tend to focus on the most recent
> > RELENG_8 build.  Then someone comes along with an older build.....
> > Hehe.  :-)
> 
> I know what you mean. Keeping up with all this "stuff" is getting harder every year. :)

So to circle back around to the tuning advice:

All of this below applies to amd64, by the way.  Yes it matters.

On 8.1-STABLE, you're going to need to tune the following settings in
/boot/loader.conf to try and gain stability.  Please note that the ZFS
ARC limiting code in 8.1-STABLE is extremely (EXTREMELY!) different than
in 8.2-STABLE; many bugs with the ARC limiting were fixed.  What I'm
saying here is that the tunings I'm about to give you will work, but you
may still see the panic happen in certain situations.  You can minimise
this chance by going to 8.2-STABLE.

Anyway, your system has 4GB of RAM installed in it, so on 8.1-STABLE I'd
recommend you try these settings:

vm.kmem_size="3584M"
vm.kmem_size_max="3584M"
vfs.zfs.arc_max="2048M"

Now, these are all chosen by me off the top of my head with absolutely
ZERO knowledge of what the memory usage on this system is like
**without** ZFS in the picture.  I'm making a lot of assumptions, and
I'm assuming worst-case scenarios.  For example, if this machine also
runs mysqld and its tuned to take up a lot of memory, I would advocate
dropping vfs.zfs.arc_max to 1536M or 1024M.  Please don't drop it "too
much"; ZFS performs best when it has lots of ARC.

Since you mentioned going to 8.2-STABLE, all you need to tune on that
version is one single tunable:

vfs.zfs.arc_max

Yup, that simple.  The kmem "stuff" was greatly improved (big thanks to
Alan Cox for that!), and some other adjustments were made (such as the
changing the default value for vm.kmem_size_scale).

HTH.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                   Mountain View, CA, US |
| Making life hard for others since 1977.               PGP 4BD6C0CB |