Panics on RELENG_6_2 (kern/111458 and kern/113823)

[Kenneth Vestergaard Schmidt]

Hello.

We're experiencing semi-regular panics on a lot of our
RELENG_6_2-boxes. We haven't exactly nailed the reason yet, but at least
the panics are consistent, indicating a semi-reproducable problem.

I noticed the two PR's mentioned in the subject, which are the same
panic and backtrace as ours.

We run RELENG_6_2 on two boxes - IBM xSeries 335/336 and Sun Fire
X2100. The IBMs have never panic'ed once, and the Suns have done so many
times, which seems to indicate that it's either and amd64-issue, or an
issue with some of the extra stuff in the X2100's kernel.

The Suns are compiled with 'machine amd64' and 'cpu HAMMER', whereas
the IBMs are 'machine i386' and 'cpu I586_CPU' + 'cpu I686_CPU'. The
Suns also has COMPAT_IA32, atpic and cpufreq, and are running powerd.

The IBMs are running with mpt(4), the Suns are running with ata(4) and
gmirror. The Suns have USB in the kernel, though it isn't used much.

We're really anxious to get this problem fixed, but I'm not sure what I
can do. I can't replicate it easily - we mostly just wait for things to
panic on their own.

We have spare boxes, so if anybody can help, we can run tests, or even
provide root-logins. I can supply any type of information needed.

For good measure, here's the kgdb output:


Unread portion of the kernel message buffer:

frame pointer           = 0x10:0x4
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 9 (thread taskq)
trap number             = 12
panic: page fault
cpuid = 1

(kgdb) bt
#0  doadump () at pcpu.h:172
#1  0x0000000000000004 in ?? ()
#2  0xffffffff80280b77 in boot (howto=260) at /usr/dana/src/freebsd6/src/sys/kern/kern_shutdown.c:409
#3  0xffffffff80281211 in panic (fmt=0xffffff001ed41980 "\bªÕ\036")
    at /usr/dana/src/freebsd6/src/sys/kern/kern_shutdown.c:565
#4  0xffffffff803cb6ef in trap_fatal (frame=0xffffff001ed41980, eva=18446742974715243016)
    at /usr/dana/src/freebsd6/src/sys/amd64/amd64/trap.c:660
#5  0xffffffff803cbc16 in trap (frame=
      {tf_rdi = 19, tf_rsi = -1098994411136, tf_rdx = 6, tf_rcx = 3221225730, tf_r8 = -1796719344, tf_r9 = -1099004634760, tf_rax = 1, tf_rbx = -1099413515464, tf_rbp = 4, tf_r10 = -2141530056, tf_r11 = 0, tf_r12 = -1098994411136, tf_r13 = 4, tf_r14 = 1, tf_r15 = 20, tf_trapno = 12, tf_addr = 396, tf_flags = -1099413515464, tf_err = 0, tf_rip = -2144901513, tf_cs = 8, tf_rflags = 65538, tf_rsp = -1796719760, tf_ss = 16}) at /usr/dana/src/freebsd6/src/sys/amd64/amd64/trap.c:238
#6  0xffffffff803b6ebb in calltrap () at /usr/dana/src/freebsd6/src/sys/amd64/amd64/exception.S:168
#7  0xffffffff80276677 in _mtx_lock_sleep (m=0xffffff0005d91338, tid=18446742974715140480, opts=6, 
    file=0xc0000102 <Address 0xc0000102 out of bounds>, line=-1796719344)
    at /usr/dana/src/freebsd6/src/sys/kern/kern_mutex.c:546
#8  0xffffffff802d589d in unp_gc (arg=0x13, pending=517216640)
    at /usr/dana/src/freebsd6/src/sys/kern/uipc_usrreq.c:1714
#9  0xffffffff802a8045 in taskqueue_run (queue=0xffffff00007d5c00)
    at /usr/dana/src/freebsd6/src/sys/kern/subr_taskqueue.c:257
#10 0xffffffff802a8d95 in taskqueue_thread_loop (arg=0x13)
    at /usr/dana/src/freebsd6/src/sys/kern/subr_taskqueue.c:376
#11 0xffffffff80267bf7 in fork_exit (callout=0xffffffff802a8d10 <taskqueue_thread_loop>, 
    arg=0xffffffff805b4e70, frame=0xffffffff94e83c50)
    at /usr/dana/src/freebsd6/src/sys/kern/kern_fork.c:821
#12 0xffffffff803b721e in fork_trampoline ()
    at /usr/dana/src/freebsd6/src/sys/amd64/amd64/exception.S:394
#13 0x0000000000000000 in ?? ()
...
#45 0x00000000007aa000 in ?? ()
#46 0xffffff001ed5aa08 in ?? ()
#47 0x0000000000000104 in ?? ()
#48 0x0000000000000000 in ?? ()
#49 0xffffff001ed5aa08 in ?? ()
#50 0xffffff001ed40720 in ?? ()
#51 0xffffffff94e83868 in ?? ()
#52 0xffffff001ed41980 in ?? ()
#53 0xffffffff80296fa6 in sched_switch (td=0xffffffff805b4e70, newtd=0x0, flags=0)
    at /usr/dana/src/freebsd6/src/sys/kern/sched_4bsd.c:973
#54 0x0000000000000000 in ?? ()
...
#125 0x0000000000000000 in ?? ()
Cannot access memory at address 0xffffffff94e84000

-- 
Kenneth Schmidt
pil.dk