System doesn't dump

[Jeremy Chadwick]

On Wed, May 29, 2013 at 08:41:38AM +0200, Dominic Fandrey wrote:
> I have a number of actions that reliably panic the system, such as
> performing shutdown -p (yes I'm booting into an inconsistent file
> system every time). Both with my notebook and my workstation.
> 
> However I cannot get the system to dump.
> 
> dumpdir=/var/crash
> and I've tried ada0s2b, /dev/ada0s2b, label/5swap, /dev/label/5swap and AUTO
> for dumpdev to no avail.
>
> The swap partition is 16g, the machines have 8g RAM and there's plenty
> of hard disk space available for /var/crash.
> 
> I'm looking for that secret, undocumented trigger, that makes the
> system dump if a panic occurs. Once upon a time dumping just worked
> if the swap partition was large enough. I miss those olden days.

Foremost: the fact you did not disclose your FreeBSD version (and SVN
rev if you have it) nor architecture is disappointing.  It matters more
than you think.  Please disclose it.

Onward ho...

If you have VGA console access, try dropping to db> and issuing the
command "call doadump" (possibly preceded by "panic").

If you have serial console access, there are ways to drop to ddb but it
depends on your kernel config (look for BREAK_TO_DEBUGGER and
ALT_BREAK_TO_DEBUGGER in /sys/conf/NOTES).  "Break" with serial, by the
way, means a serial-level break signal (often why I prefer
ALT_BREAK_TO_DEBUGGER).

After doing "call doadump" you should definitely see the kernel dumping
memory to swap (it gives a progress indicator of sorts).  Google for the
phrase "call doadump" and look at some of the results to get an idea of
what the output normally is during that phase, for comparison.

If you don't see such, I'm sure many of the kernel folks here can help
figure out why.

See sysctl debug.ddb.scripting.scripts for what should get automatically
done on a panic.  This may or may not be affected by ddb_enable="yes" in
rc.conf (which mandates DDB being enabled in your kernel) -- I can't
remember though, so someone else may want to comment.

If your issue is that the kernel actually *does* dump memory to swap but
that on boot-up savecore(8) doesn't recover the memory dump and populate
relevant files in /var/crash: that's a separate issue that has been
discussed for probably 10 years or longer with (to my knowledge) no
definitive explanation.  Theories presented (going off of memory here)
were that that something ended up writing over parts of the "panic
metadata" on the swap disk/slice/etc. and thus savecore(8) finds
nothing.  This is why rc scripts/etc. have to make sure to look for the
swap "panic metadata" and run savecore(8) **before** issuing dumpon(8).

My opinion, others' may vary:

Stick with using dumpdev="auto" in rc.conf, assuming you have a
/etc/fstab entry of "swap" somewhere.  Swap should ideally be a
partition or slice, not something abstracted out by other layers (see
above paragraph for why I advocate that, but my additional opinion is
that when it comes to getting a kernel dump and system configurations,
KISS principle applies heavily.  If your system is crashing, the last
thing you want to deal with is why you can't get a kernel dump -- you
could spend more time doing that than you do getting the panic info +
debugging the actual crash), but again, this is my own opinion and there
are legitimate other opinions as well -- I just follow what I do because
I know it works.

Likewise I always get wary of people's setups when I start seeing
labels mentioned.  *waves cane*  Screw all this newfandangled stuff.
:-)

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Mountain View, CA, US                                            |
| Making life hard for others since 1977.             PGP 4BD6C0CB |