How to troubleshoot solid freeze-up?
rwatson at FreeBSD.org
Mon Oct 3 03:49:58 PDT 2005
On Sun, 2 Oct 2005, David S. Madole wrote:
> I'm looking for some tips on how to troubleshoot a possible driver
> problem. Here is the scenario:
> 1. Using a Pentium II 333Mhz mobile processor, 82443BX motherboard, and
> Intel i82559 NIC (fxp driver).
> 2. A combination of heavy disk I/O, high CPU utilization, and high
> network traffic causes a solid machine freeze-up sometime between 10
> minutes and 3 hours of running.
> 3. Replacing the NIC with a DP83815-based card (sis driver) seems to
> solve the problem. I have run the problem load for up to 8 hours without
> issue on this NIC.
> 4. The problem is reproducable on multiple identical machines with
> multiple identical NICs. Also reproducable on an i82558 NIC integrated
> on the motherboard.
> How can I go about collecting useful information to troubleshoot this
> when the machine locks solid? How can a get a core under this scenario?
> Switching to another NIC permanently is not a great solution because
> this is a semi-embedded application and I need to use the NIC on the
The normal method is to use a break signal to get into the debugger.
Depending on your hardware and software configuration, this may be more or
less easy. First, you'll need to configure options BREAK_TO_DEBUGGER into
your kernel. You can break into the debugger in one of three ways:
(1) Ctrl-alt-esc on a syscons console. Note that because the syscons
console is under the Giant lock, reliability of this mechanism to get
into the debugger on FreeBSD 5.x is reduced. It is quite a bit better
in 6.x, and will continue to get better as the use of Giant is
reduced. If this doesn't work for you, try (2).
(2) Serial break on the first serial console port. Because the sio driver
uses a fast interrupt handler, this is quite a reliable way to get
into the debugger unless interrupts are disabled, in which case the
serial port can't interrupt the CPU to drop into the debugger. If
this doesn't work for you, try (3).
(3) Break to debugger using an NMI. Some hardware, especially evaluation
hardware, comes with an NMI button, frob, or other way to initiate a
drop to the debugger despite interrupts being disabled. Hardware
watchdogs are often also able to generate an NMI.
I find that, except in pretty exceptional circumstances, (2) works quite
well. You can find a section on kernel debugging in the FreeBSD handbook;
my general advice is to compile in KDB, DDB, BREAK_TO_DEBUGGER, WITNESS,
and INVARIANTS, and see where that gets you using a serial console. DDB
is pretty easy to use for basic debugging -- i.e., checking thread state,
checking lock state, generating stack traces, etc. Depending on the bug,
you might also need/want to use kgdb via serial or on a core dump.
Robert N M Watson
More information about the freebsd-hackers