How to troubleshoot solid freeze-up?

[Robert Watson]

On Sun, 2 Oct 2005, David S. Madole wrote:

> I'm looking for some tips on how to troubleshoot a possible driver 
> problem. Here is the scenario:
>
> 1. Using a Pentium II 333Mhz mobile processor, 82443BX motherboard, and 
> Intel i82559 NIC (fxp driver).
>
> 2. A combination of heavy disk I/O, high CPU utilization, and high 
> network traffic causes a solid machine freeze-up sometime between 10 
> minutes and 3 hours of running.
>
> 3. Replacing the NIC with a DP83815-based card (sis driver) seems to 
> solve the problem. I have run the problem load for up to 8 hours without 
> issue on this NIC.
>
> 4. The problem is reproducable on multiple identical machines with 
> multiple identical NICs. Also reproducable on an i82558 NIC integrated 
> on the motherboard.
>
> How can I go about collecting useful information to troubleshoot this 
> when the machine locks solid? How can a get a core under this scenario?
>
> Switching to another NIC permanently is not a great solution because 
> this is a semi-embedded application and I need to use the NIC on the 
> motherboard.

The normal method is to use a break signal to get into the debugger. 
Depending on your hardware and software configuration, this may be more or 
less easy.  First, you'll need to configure options BREAK_TO_DEBUGGER into 
your kernel.  You can break into the debugger in one of three ways:

(1) Ctrl-alt-esc on a syscons console.  Note that because the syscons
     console is under the Giant lock, reliability of this mechanism to get
     into the debugger on FreeBSD 5.x is reduced.  It is quite a bit better
     in 6.x, and will continue to get better as the use of Giant is
     reduced.  If this doesn't work for you, try (2).

(2) Serial break on the first serial console port.  Because the sio driver
     uses a fast interrupt handler, this is quite a reliable way to get
     into the debugger unless interrupts are disabled, in which case the
     serial port can't interrupt the CPU to drop into the debugger.  If
     this doesn't work for you, try (3).

(3) Break to debugger using an NMI.  Some hardware, especially evaluation
     hardware, comes with an NMI button, frob, or other way to initiate a
     drop to the debugger despite interrupts being disabled.  Hardware
     watchdogs are often also able to generate an NMI.

I find that, except in pretty exceptional circumstances, (2) works quite 
well.  You can find a section on kernel debugging in the FreeBSD handbook; 
my general advice is to compile in KDB, DDB, BREAK_TO_DEBUGGER, WITNESS, 
and INVARIANTS, and see where that gets you using a serial console.  DDB 
is pretty easy to use for basic debugging -- i.e., checking thread state, 
checking lock state, generating stack traces, etc.  Depending on the bug, 
you might also need/want to use kgdb via serial or on a core dump.

Robert N M Watson