svn commit: r222537 - in head/sys: kern sys

[mdf at FreeBSD.org]

On Tue, May 31, 2011 at 2:46 PM, Kenneth D. Merry <ken at freebsd.org> wrote:
> On Tue, May 31, 2011 at 14:00:18 -0700, mdf at FreeBSD.org wrote:
>> On Tue, May 31, 2011 at 10:29 AM, Kenneth D. Merry <ken at freebsd.org> wrote:
>> > Author: ken
>> > Date: Tue May 31 17:29:58 2011
>> > New Revision: 222537
>> > URL: http://svn.freebsd.org/changeset/base/222537
>> >
>> > Log:
>> > ?Fix apparent garbage in the message buffer.
>> >
>> > ?While we have had a fix in place (options PRINTF_BUFR_SIZE=128) to fix
>> > ?scrambled console output, the message buffer and syslog were still getting
>> > ?log messages one character at a time. ?While all of the characters still
>> > ?made it into the log (courtesy of atomic operations), they were often
>> > ?interleaved when there were multiple threads writing to the buffer at the
>> > ?same time.
>>
>> This seems to panic my box with "lock "msgbuf" 0xfffffe0127ffffe0
>> already initialized".
>>
>> Unfortunately, though I booted with a fresh CURRENT this morning
>> successfully, both /boot/kernel and /boot/kernel.old give this panic.
>> To add insult to injury, when the kernel drops into the debugger, my
>> keyboard input no longer works so I can't get a stack, etc.
>
> Uh-oh!
>
>> So:
>>
>> 1) Is there anything else I can do to help debug this?
>> 2) how can I resurrect this box without a reinstall?
>>
>> I will try to repro on a virtual machine so I have a snapshot to come back to.
>
> My guess is that this is an issue with the message buffer reinitialization
> path.  lock_init() (called by mtx_init()) has an assert to make sure that
> the lock is initialized, and that is just a flag check.
>
> Since the spin lock is part of the message buffer structure, if it is held
> over from a previous boot, the LO_INITIALIZED flag may still be set.
>
> Try power cycling the machine.  If it is an issue with re-initialization,
> that should clear the memory and allow you to boot.

Hmm, apparently my previous presses of the power button weren't long
enough.  I let it sit off for 20 seconds and it boots okay now.

Thanks,
matthew

> My testing has been with VMs (under Xen), so the reinit path has probably
> not been tested as fully as it should have been.  Sorry about that!
>
> As for the debugger, that's another issue altogether.  It does work for me,
> but then again if the spin lock initialization is broken for the message
> buffer that may affect things.
>
> Try a cold boot and see if that helps.  If so, I think we can probably just
> bzero the mutex in msgbuf_reinit() and that will fix things.
>
> Ken
> --
> Kenneth Merry
> ken at FreeBSD.ORG
>