svn commit: r222537 - in head/sys: kern sys

[Kenneth D. Merry]

On Tue, May 31, 2011 at 14:00:18 -0700, mdf at FreeBSD.org wrote:
> On Tue, May 31, 2011 at 10:29 AM, Kenneth D. Merry <ken at freebsd.org> wrote:
> > Author: ken
> > Date: Tue May 31 17:29:58 2011
> > New Revision: 222537
> > URL: http://svn.freebsd.org/changeset/base/222537
> >
> > Log:
> > ?Fix apparent garbage in the message buffer.
> >
> > ?While we have had a fix in place (options PRINTF_BUFR_SIZE=128) to fix
> > ?scrambled console output, the message buffer and syslog were still getting
> > ?log messages one character at a time. ?While all of the characters still
> > ?made it into the log (courtesy of atomic operations), they were often
> > ?interleaved when there were multiple threads writing to the buffer at the
> > ?same time.
> 
> This seems to panic my box with "lock "msgbuf" 0xfffffe0127ffffe0
> already initialized".
> 
> Unfortunately, though I booted with a fresh CURRENT this morning
> successfully, both /boot/kernel and /boot/kernel.old give this panic.
> To add insult to injury, when the kernel drops into the debugger, my
> keyboard input no longer works so I can't get a stack, etc.

Uh-oh!

> So:
> 
> 1) Is there anything else I can do to help debug this?
> 2) how can I resurrect this box without a reinstall?
> 
> I will try to repro on a virtual machine so I have a snapshot to come back to.

My guess is that this is an issue with the message buffer reinitialization
path.  lock_init() (called by mtx_init()) has an assert to make sure that
the lock is initialized, and that is just a flag check.

Since the spin lock is part of the message buffer structure, if it is held
over from a previous boot, the LO_INITIALIZED flag may still be set.

Try power cycling the machine.  If it is an issue with re-initialization,
that should clear the memory and allow you to boot.

My testing has been with VMs (under Xen), so the reinit path has probably
not been tested as fully as it should have been.  Sorry about that!

As for the debugger, that's another issue altogether.  It does work for me,
but then again if the spin lock initialization is broken for the message
buffer that may affect things.

Try a cold boot and see if that helps.  If so, I think we can probably just
bzero the mutex in msgbuf_reinit() and that will fix things.

Ken
-- 
Kenneth Merry
ken at FreeBSD.ORG