panic: stable/10 with Debugging enabled in kern_cons.c:500

[Konstantin Belousov]

On Sun, Jan 12, 2014 at 01:10:37PM -0800, Sean Bruno wrote:
> On Sun, 2014-01-12 at 05:42 +0200, Konstantin Belousov wrote:
> > On Sat, Jan 11, 2014 at 02:44:57PM -0800, Sean Bruno wrote:
> > > On Sat, 2014-01-11 at 14:12 -0800, Sean Bruno wrote:
> > > > I can't imagine that I'm the first person to run with the following
> > > > debug options enabled:
> > > > 
> > > > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html
> > > > 
> > > > But, I can't get stable/10 to post on a plain old supermicro box with
> > > > those debugging symbols enabled.  
> > > > 
> > > > 
> > > > random: unblocking device.
> > > > panic: mtx_lock_spin: recursed on non-recursive mutex cnputs_mtx
> > > > @ /usr/src/sys/kern/kern_cons.c:500
> > > > 
> > > > cpuid = 0
> > > > KDB: stack backtrace:
> > > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> > > > 0xfffffe1764e681b0
> > > > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe1764e68260
> > > > vpanic() at vpanic+0x126/frame 0xfffffe1764e682a0
> > > > kassert_panic() at kassert_panic+0x136/frame 0xfffffe1764e68310
> > > > __mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x166/frame
> > > > 0xfffffe1764e68350
> > > > cnputs() at cnputs+0x32/frame 0xfffffe1764e68370
> > > > putchar() at putchar+0x13a/frame 0xfffffe1764e683f0
> > > > kvprintf() at kvprintf+0xda/frame 0xfffffe1764e684f0
> > > > vprintf() at vprintf+0x87/frame 0xfffffe1764e685c0
> > > > printf() at printf+0x43/frame 0xfffffe1764e68620
> > > > witness_checkorder() at witness_checkorder+0xa99/frame
> > > > 0xfffffe1764e686b0
> > > > __mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x95/frame
> > > > 0xfffffe1764e686f0
> > > > uart_cnputc() at uart_cnputc+0x3b/frame 0xfffffe1764e68710
> > > > cnputc() at cnputc+0x7f/frame 0xfffffe1764e68740
> > > > cnputs() at cnputs+0x58/frame 0xfffffe1764e68760
> > > > putchar() at putchar+0x13a/frame 0xfffffe1764e687e0
> > > > kvprintf() at kvprintf+0xda/frame 0xfffffe1764e688e0
> > > > vprintf() at vprintf+0x87/frame 0xfffffe1764e689b0
> > > > printf() at printf+0x43/frame 0xfffffe1764e68a10
> > > > witness_checkorder() at witness_checkorder+0xa99/frame
> > > > 0xfffffe1764e68aa0
> > > > __mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x95/frame
> > > > 0xfffffe1764e68ae0
> > > > msleep_spin_sbt() at msleep_spin_sbt+0x90/frame 0xfffffe1764e68b70
> > > > random_kthread() at random_kthread+0x1d0/frame 0xfffffe1764e68bb0
> > > > fork_exit() at fork_exit+0x84/frame 0xfffffe1764e68bf0
> > > > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1764e68bf0
> > > > --- trap 0, rip = 0, rsp = 0xfffffe1764e68cb0, rbp = 0 ---
> > > > KDB: enter: panic
> > > > [ thread pid 14 tid 100057 ]
> > > > Stopped at      kdb_enter+0x3e: movq    $0,kdb_why
> > > 
> > > 
> > > Witness induced.  Removed the witness option and this doesn't happen
> > > (obviously).  Can I get someone familiar with console code to look into
> > > this?  Kind of crappy when I'm investigating a resource starvation
> > > issue.
> > 
> > Add
> > options  	WITNESS_SKIPSPIN
> > to the kernel config.  This is required now, unfortunately.
> 
> 
> Confirmed.  Thank you.
> 
> It looks like the machine becomes seriously unuseable with this much
> debugging on.  After a few hours of testing.  I cannot run "svn up" any
> more and the system seems sluggish.
> 
> root at redbuild03:/usr/src # svn up -r 260075
> Updating '.':
> load: 2.06  cmd: svn 12480 [*kmem arena] 624.87r 0.35u 434.25s 0% 11672k
> load: 2.06  cmd: svn 12480 [*kmem arena] 625.71r 0.35u 434.25s 0% 11672k
> load: 2.06  cmd: svn 12480 [*kmem arena] 625.91r 0.35u 434.25s 0% 11672k
> load: 2.06  cmd: svn 12480 [*kmem arena] 626.09r 0.35u 434.25s 0% 11672k
> load: 2.06  cmd: svn 12480 [*kmem arena] 626.27r 0.35u 434.25s 0% 11672k
> load: 2.06  cmd: svn 12480 [*kmem arena] 626.45r 0.35u 434.25s 0% 11672k
> load: 2.06  cmd: svn 12480 [*kmem arena] 626.61r 0.35u 434.25s 0% 11672k
> load: 2.06  cmd: svn 12480 [*kmem arena] 626.79r 0.35u 434.25s 0% 11672k
> load: 2.06  cmd: svn 12480 [*kmem arena] 626.95r 0.35u 434.25s 0% 11672k
> load: 2.06  cmd: svn 12480 [*kmem arena] 627.13r 0.35u 434.25s 0% 11672k
> 
> 
> Is this a symptom of the level of debugging or am I starting to come
> close to a problem that needs investigation?

This is a DIAGNOSTIC option, I think.  Set sysctl debug.enable_vmem_check=0;
I thought that the default was changed in HEAD long time ago, but apparently
it was not.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20140112/b58002d2/attachment.sig>