FreeBSD 6.1 Instability

[John Baldwin]

On Thursday 05 October 2006 15:44, Charles Ulrich wrote:
> Greetings,
> 
> We have been running FreeBSD on our mail servers for about as long as I can 
> remember. Recently, we decided to go SMP to handle increased mail load. After 
> assembling the hardware, installing the OS and software, and restoring all of 
> our data, we noticed in testing that our first machine began hanging 
> semi-regularly when it began processing lots of mail. Disabling SMP 
> eliminated the hangs completely. We tried it all again on completely 
> different hardware with exactly the same result. Our conclusion: somethings's 
> buggy in SMP.
> 
> Here are the symptoms. The machine hangs, and becomes completely 
> unresponsive.  It looks like a deadlock.  It will sometimes respond to the 
> power button and shut down (without being able to first sync and unmount 
> filesystems), and sometimes the power button event gets caught in the 
> deadlock.  Sinceit's not actually a crash, there is no core dump or other 
> debugging information. In the most recent situation, it hung at different 
> points every time I tried to compile ezm3, after successfully compiling other 
> packages.
> 
> We're system administrators, not kernel hackers, so this is a plea for help. I 
> wouldn't know where to start, but I'm hoping someone can point me in the 
> right direction. We're also willing to give a (trustworthy) FreeBSD developer 
> root access to the test machine since it's just sitting idle right now. If 
> you need to crash it, that's fine. We'll have people during normal business 
> hours who know how to push a reset button.
> 
> Thanks for your time.

Compile a debug kernel and include 'DDB' in the kernel.  When it hangs, break into
the debugger and type 'panic' to have it panic the machine and write out a crash
dump.  Once you have the crash dump, download http://www.FreeBSD.org/~jhb/gdb/gdb6
and do this:

$ kgdb /usr/obj/usr/src/sys/FOO/kernel.debug /var/crash/vmcore.X

(where FOO is your kernel config file and X is the right vmcore file)

Then do this:

(gdb) source /path/to/gdb6
(gdb) ps
...

And reply with the output from the 'ps' command.

-- 
John Baldwin