US-III crashes on current

[Florian Smeets]

On 21.04.09 20:58, Marius Strobl wrote:
> On Tue, Apr 21, 2009 at 01:45:27AM +0200, Florian Smeets wrote:
>>
>> Yes, i can still reproduce this on every shutdown. Tried with r191337.
>> Trace is still the same.
>>
>
> Could you please run gdb(1) on the corresponding kernel.debug
> and report the output of the following commands?
> l *(0xc034c96c)
> l *(callout_lock+0x40)
> Change as needed if the addresses differ from the above
> backtrace. Hrm, the one you reported to scsi@ actually
> is a bit different:
>> -- fast data access mmu miss tar=0x1454156000 %o7=0xc040e7a4 --
>> _mtx_lock_spin_flags() at _mtx_lock_spin_flags+0x5c
>> callout_lock() at callout_lock+0x50
>
> In that case please additionally get the output of
> l *(_mtx_lock_spin_flags+0x5c)
>

OK, to get this straight this is the trace I'm talking about.

Uptime: 19h19m49s
panic: trap: fast data access mmu miss
cpuid = 0
KDB: enter: panic
[thread pid 97473 tid 100179 ]
Stopped at      kdb_enter+0x80: ta              %xcc, 1
db> where
Tracing pid 97473 tid 100179 td 0xfffff80006dfc370
panic() at panic+0x20c
trap() at trap+0x4d0
-- fast data access mmu miss tar=0x20007e000 %o7=0xc03f70a4 --
callout_lock() at callout_lock+0x20
untimeout() at untimeout+0xc
isp_done() at isp_done+0x140
isp_intr() at isp_intr+0x3eb8
isp_poll() at isp_poll+0x38
xpt_polled_action() at xpt_polled_action+0xc8
dashutdown() at dashutdown+0x16c
boot() at boot+0x850
reboot() at reboot+0x64
syscall() at syscall+0x2b4
-- syscall (55, FreeBSD ELF64, reboot) %o7=0x1013e4 --
userland() at 0x40564948
user trace: trap %o7=0x1013e4
pc 0x40564948, sp 0x7fdffffe201
pc 0x100df0, sp 0x7fdffffe2c1
pc 0x40206954, sp 0x7fdffffe381
done

(gdb) l *(0xc03f70a4)
0xc03f70a4 is in spinlock_exit (/usr/src/sys/sparc64/sparc64/machdep.c:232).
227	spinlock_exit(void)
228	{
229		struct thread *td;
230	
231		td = curthread;
232		critical_exit();
233		td->td_md.md_spinlock_count--;
234		if (td->td_md.md_spinlock_count == 0)
235			wrpr(pil, td->td_md.md_saved_pil, 0);
236	}
(gdb) l *(callout_lock+0x20)
0xc0225000 is in callout_lock (/usr/src/sys/kern/kern_timeout.c:270).
265	{
266		struct callout_cpu *cc;
267		int cpu;
268	
269		for (;;) {
270			cpu = c->c_cpu;
271			cc = CC_CPU(cpu);
272			CC_LOCK(cc);
273			if (cpu == c->c_cpu)
274				break;


I had witness in my kernel when i reported this to scsi, i don't have 
that turned on right now, so perhaps this is why the other trace 
included _mtx_lock_spin_flags, it's not in the trace now but if i run 
the command you requested i get something non the less.

(gdb) l *(_mtx_lock_spin_flags+0x5c)
0xc01ff2bc is in _mtx_lock_spin_flags (/usr/src/sys/kern/kern_mutex.c:225).
220			KASSERT((m->lock_object.lo_flags & LO_RECURSABLE) != 0,
221		    ("mtx_lock_spin: recursed on non-recursive mutex %s @ %s:%d\n",
222			    m->lock_object.lo_name, file, line));
223		WITNESS_CHECKORDER(&m->lock_object, opts | LOP_NEWORDER | 
LOP_EXCLUSIVE,
224		    file, line, NULL);
225		_get_spin_lock(m, curthread, opts, file, line);
226		LOCK_LOG_LOCK("LOCK", &m->lock_object, opts, m->mtx_recurse, file,
227		    line);
228		WITNESS_LOCK(&m->lock_object, opts | LOP_EXCLUSIVE, file, line);
229	}

Thanks for looking at this!

Cheers,
Florian