Automatic reboot doesn't reboot

[Jeremy Chadwick]

On Mon, May 02, 2011 at 04:32:30PM +0200, Olaf Seibert wrote:
> I have a FreeBSD/amd64 8.2 server that has a few ZFS file systems served
> over NFS.  It has 8 GB of memory. There are 6 disks of 1,5 TB each
> forming a pool with raidz2.
> 
> >From time to time it crashes with some stack backtrace (included below).
> This already happened before the upgrade to 8.2.
> 
> Now a crash of a file server is annoying, but if it reboots
> automatically, there is just a few minutes of downtime (most of it is
> even spent by the BIOS before it gets to boot the OS).
> 
> However, it doesn't automatically reboot in 15 seconds, as promised.
> It just sits there the whole weekend, until I log onto the IPMI console
> and press the virtual reset button.

There are two things you might try fiddling with.  These are sysctls so
you can try them on the fly:

hw.acpi.disable_on_reboot
hw.acpi.handle_reboot

On our systems we set hw.acpi.handle_reboot=1 to speed up the reboot
process.  I remember hearing long ago how some people had issues getting
their machines to reboot (sometimes 100% of the time, other times
occasionally); using ACPI to reboot the machine fixed their issues.

> This was visible before I did that (4-finger copy):
> 
> panic: kmem_alloc(131072): kmem_map too small: 3428782080 total allocated
> cpuid = 0

Check out the thread Peter Jeremy provided.  This is a near-sure
indicator of ZFS ARC exhaustion, and you seem to know of that.  What's
very interesting to me is this part of your mail:

> There is some tuning in /boot/loader.conf from previous attempts tune to
> avoid crashes.
> 
> vm.kmem_size="16G"
> vfs.zfs.arc_max="4G"
> 
> Is that still useful, or does it harm by now? Real memory is 8 GB.
> I note that if I look with sysctl, I see
> 
> vm.kmem_size: 3739230208
> vfs.zfs.arc_max: 2665488384
> 
> which doesn't seem to match these attempted settings.

Is this box running i386 or amd64?  If amd64, I can't explain why your
/boot/loader.conf settings aren't taking -- they should be for sure.
Maybe provide us a full dmesg and XXX out things you consider
sensitive.  If i386, I'm not too surprised that some automatic defaults
get chosen instead of what you ask.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |