Yet another crash - help with interpreting dump

[Andrea Venturoli]

Hello.

A server of mine hanged three days ago. As the office reopened I had 
someone press Ctrl-Alt-Esc and type "panic".

This is the dump I obtained:


> # kgdb kernel.debug /var/crash/vmcore.0
> [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd".
> 
> Unread portion of the kernel message buffer:
> 
> 
> #0  doadump () at pcpu.h:172
> 172             __asm __volatile("movq %%gs:0,%0" : "=r" (td));
> (kgdb) bt
> #0  doadump () at pcpu.h:172
> #1  0xffffffff80232879 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
> #2  0xffffffff8023230b in panic (fmt=0xffffffff80394189 "from debugger") at /usr/src/sys/kern/kern_shutdown.c:565
> #3  0xffffffff80177272 in db_panic (addr=0, have_addr=0, count=0, modif=0x0) at /usr/src/sys/ddb/db_command.c:438
> #4  0xffffffff801777b5 in db_command_loop () at /usr/src/sys/ddb/db_command.c:350
> #5  0xffffffff801796ad in db_trap (type=-1315342064, code=0) at /usr/src/sys/ddb/db_main.c:222
> #6  0xffffffff8024fbb9 in kdb_trap (type=3, code=0, tf=0xffffffffb1997a10) at /usr/src/sys/kern/subr_kdb.c:473
> #7  0xffffffff80349f14 in trap (frame=
>       {tf_rdi = 0, tf_rsi = -2139025408, tf_rdx = 1, tf_rcx = 1177845, tf_r8 = 1048064, tf_r9 = 10, tf_rax = 38, tf_rbx = 0, tf_rbp = -1315341616, tf_r10 = -1315341856, tf_r11 = 4294967256, tf_r12 = -2141325600, tf_r13 = -1098198600192, tf_r14 = 2, tf_r15 = 0, tf_trapno = 3, tf_addr = 0, tf_flags = -2145099097, tf_err = 0, tf_rip = -2145061233, tf_cs = 8, tf_rflags = 646, tf_rsp = -1315341616, tf_ss = 16}) at /usr/src/sys/amd64/amd64/trap.c:442
> #8  0xffffffff8033558b in calltrap () at /usr/src/sys/amd64/amd64/exception.S:168
> #9  0xffffffff8024f68f in kdb_enter (msg=0x0) at cpufunc.h:63
> #10 0xffffffff8036650e in scgetc (sc=0xffffffff805df6e0, flags=2) at /usr/src/sys/dev/syscons/syscons.c:3365
> #11 0xffffffff803667a3 in sckbdevent (thiskbd=0xffffff0000912400, event=-2139025408, arg=0x1) at /usr/src/sys/dev/syscons/syscons.c:659
> #12 0xffffffff801b97bd in kbdmux_intr (kbd=0xffffff0000912400, arg=0xffffffff80811000) at /usr/src/sys/dev/kbdmux/kbdmux.c:548
> #13 0xffffffff801b8e40 in kbdmux_kbd_intr (xkbd=0x0, pending=-2139025408) at /usr/src/sys/dev/kbdmux/kbdmux.c:199
> #14 0xffffffff80258675 in taskqueue_run (queue=0xffffff0000948000) at /usr/src/sys/kern/subr_taskqueue.c:257
> #15 0xffffffff8021b5bc in ithread_loop (arg=0xffffff0000049380) at /usr/src/sys/kern/kern_intr.c:682
> #16 0xffffffff8021a30b in fork_exit (callout=0xffffffff8021b470 <ithread_loop>, arg=0xffffff0000049380, frame=0xffffffffb1997c50) at /usr/src/sys/kern/kern_fork.c:821
> #17 0xffffffff803358ee in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:394
> #18 0x0000000000000000 in ?? ()
> #19 0x0000000000000000 in ?? ()
> #20 0x0000000000000001 in ?? ()
> #21 0x0000000000000000 in ?? ()
> #22 0x0000000000000000 in ?? ()
> #23 0x0000000000000000 in ?? ()
> #24 0x0000000000000000 in ?? ()
> #25 0x0000000000000000 in ?? ()
> #26 0x0000000000000000 in ?? ()
> #27 0x0000000000000000 in ?? ()
> #28 0x0000000000000000 in ?? ()
> #29 0x0000000000000000 in ?? ()
> #30 0x0000000000000000 in ?? ()
> #31 0x0000000000000000 in ?? ()
> #32 0x0000000000000000 in ?? ()
> #33 0x0000000000000000 in ?? ()
> #34 0x0000000000000000 in ?? ()
> #35 0x0000000000000000 in ?? ()
> #36 0x0000000000000000 in ?? ()
> #37 0x0000000000000000 in ?? ()
> #38 0x0000000000000000 in ?? ()
> #39 0x0000000000000000 in ?? ()
> #40 0x0000000000000000 in ?? ()
> #41 0x0000000000000000 in ?? ()
> #42 0x0000000000000000 in ?? ()
> #43 0x0000000000000000 in ?? ()
> #44 0x0000000000000000 in ?? ()
> #45 0x0000000000000000 in ?? ()
> #46 0x0000000000000000 in ?? ()
> #47 0x0000000000000000 in ?? ()
> #48 0x0000000000000000 in ?? ()
> #49 0x0000000000000000 in ?? ()
> #50 0x00000000007cc000 in ?? ()
> #51 0xffffffff00000001 in ?? ()
> #52 0x0000000000000001 in ?? ()
> #53 0xffffff007a85c000 in ?? ()
> #54 0xffffff007a87d980 in ?? ()
> #55 0xffffffffb1997b80 in ?? ()
> #56 0xffffffffb1997b58 in ?? ()
> #57 0xffffff007a840720 in ?? ()
> #58 0xffffffff802472aa in sched_switch (td=0xffffff0000049380, newtd=0xffffffff8021b470, flags=0) at /usr/src/sys/kern/sched_4bsd.c:973
> Previous frame inner to this frame (corrupt stack?)

I have no message in the logs even if I have the following options in my 
kernel conf:

 > options         KDB
 > options         DDB
 > options         KDB_UNATTENDED
 > options         INVARIANTS
 > options         INVARIANT_SUPPORT
 > options         WITNESS
 > options         DEBUG_LOCKS
 > options         DEBUG_VFS_LOCKS
 > options         DIAGNOSTIC



The box is running 6.2-p1/amd64 with amr driver from 6.0, since later 
versions hangs continuously.

Any hint? Even some help in getting more info would be useful.
I can have someon type commands in DDB (like he did with panic), but is 
their output saved somewhere?

  bye & Thanks
	av.