new panic in cpu_reset() with WITNESS

[mdf at FreeBSD.org]

2012/1/17 Gleb Smirnoff <glebius at freebsd.org>:
>  New panic has been introduced somewhere between
> r229851 and r229932, that happens on shutdown if
> kernel has WITNESS and doesn't have WITNESS_SKIPSPIN.
>
> Uptime: 1h0m17s
> Rebooting...
> panic: mtx_lock_spin: recursed on non-recursive mutex cnputs_mtx @ /usr/src/head/sys/kern/kern_cons.c:500
> cpuid = 0
> KDB: enter: panic
> [ thread pid 1 tid 100001 ]
> Stopped at      kdb_enter+0x3b: movq    $0,0x514d32(%rip)
> db>
> db> bt
> Tracing pid 1 tid 100001 td 0xfffffe0001d5e000
> kdb_enter() at kdb_enter+0x3b
> panic() at panic+0x1c7
> _mtx_lock_spin_flags() at _mtx_lock_spin_flags+0x10f
> cnputs() at cnputs+0x7a
> putchar() at putchar+0x11f
> kvprintf() at kvprintf+0x83
> vprintf() at vprintf+0x85
> printf() at printf+0x67
> witness_checkorder() at witness_checkorder+0x773
> _mtx_lock_spin_flags() at _mtx_lock_spin_flags+0x99
> uart_cnputc() at uart_cnputc+0x3e
> cnputc() at cnputc+0x4c
> cnputs() at cnputs+0x26
> putchar() at putchar+0x11f
> kvprintf() at kvprintf+0x83
> vprintf() at vprintf+0x85
> printf() at printf+0x67
> cpu_reset() at cpu_reset+0x81
> kern_reboot() at kern_reboot+0x3a5
> --More--^M        ^Msys_reboot() at sys_reboot+0x42
> amd64_syscall() at amd64_syscall+0x39e
> Xfast_syscall() at Xfast_syscall+0xf7
> --- syscall (55, FreeBSD ELF64, sys_reboot), rip = 0x40ea3c, rsp = 0x7fffffffd6d8, rbp = 0x49 ---
> db>
> db> show locks
> exclusive sleep mutex Giant (Giant) r = 0 (0xffffffff809bc560) locked @ /usr/src/head/sys/kern/kern_module.c:101
> exclusive spin mutex smp rendezvous (smp rendezvous) r = 0 (0xffffffff80a08840) locked @ /usr/src/head/sys/kern/kern_shutdown.c:542
> db>
>
> So the problem is that we are holding smp rendezvous mutex during the cpu_reset().
> No mutexes should be obtained after it. However, since cpu_reset() does priting
> we obtain cnputs_mtx, and later obtain uart_hwmtx. The latter is hardcoded in
> the subr_witness.c as mutex to obtain before smp rendezvous, this triggers
> yet another printf from witness, that finally panics due to recursing on
> cnputs_mtx.

At $WORK we explicitly marked cnputs_mtx as NO_WITNESS since it didn't
seem possible to fit it into the heirarchy in any sane way, since a
print can come from basically anywhere.

If anyone has a better fix, that'd be great, but I haven't been able
to think of one.

Thanks,
matthew