Automatic reboot doesn't reboot

[Jeremy Chadwick]

On Tue, May 03, 2011 at 02:30:15PM +0200, Olaf Seibert wrote:
> On Tue 03 May 2011 at 05:20:52 -0700, Jeremy Chadwick wrote:
> > To be on the safe side, pick something that's small at first, then work
> > your way up.  You'll need probably 1+ weeks of heavy ZFS I/O between
> > tests (e.g. don't change the tunable, reboot, then 4 hours later declare
> > the new (larger) value as stable).
> 
> Ah, that's important: so far it seemed to me that a *too small* value
> (for all various tunables) would cause problems, but now you're saying
> that *too large* is the problem (at least for vfs.zfs.arc_max)! 

Too small = not-so-great performance (less data in the ARC means more
reads from the disks.  Disks are slower than RAM :-) ).

Too large = increased risk of kmem exhaustion panic.

> This machine has mixed loads; from time to time somebody starts a big
> job with lots of I/O, and in between it is much more modestly loaded.

I would recommend starting small (maybe 1/3rd of your physical RAM?) and
increase from there.  You can try the opposite technique too -- start
large (e.g. 3/4ths of RAM) and wait for a panic.  I'm of the opinion
that I'd rather have a stable system with less memory used for ARC than
a system which could panic and have more memory for ARC.

Sadly there's no 100% reliable way to calculate what's "ideal".  For
example I might use a smaller value than 6144M on a machine where mysqld
is tuned to utilise lots of RAM.  There's a balancing act that goes on
that takes some time to figure out.

For example, on our FreeBSD ZFS-backed NFS filer on our network, I ran
with a 3/4th amount for quite some time (we're talking 4-5 months).
Then suddenly one day I noticed the client machines were complaining
about NFS timeouts, etc...  Got on the filer, lo and behold kmem
exhaustion.  I decreased arc_max by about 1024M and it's been fine
since.

There's a lot of evolution that's occurred in the FreeBSD ZFS kernel
code over the years too.  Originally arc_max was a "high-water mark" of
some sort, but code was changed to make it a hard limit as much as it
could be.  Then some edge cases were found where it could still exceed
the maximum, so those were fixed, etc...  Tracking all the changes is
very difficult (I became very frustrated/irate at having to do so,
wishing that there was more of a "state of ZFS" announcement sent out
every so often so users/admins would know what's changed and adjust
things appropriately), requiring an admin to follow commits.  That's
just the nature of the beast.

> > So for example on an 8GB RAM machine, I might recommend starting with
> > vfs.zfs.arc_max="4096M" and let that run for a while.  If you find your
> > "Wired" value in top(1) remains fairly constant after a week or so of
> > heavy I/O, consider bumping up the value a bit more (say 4608M).
> 
> I'll do just that.

Let us know how things turn out.  Follow-ups that indicate things are
working are just as important as initial mails stating things aren't,
especially if you're someone searching the Web to try and find an answer
to what this kmem thing is all about.....  :-)

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |