svn commit: r222537 - in head/sys: kern sys

[mdf at FreeBSD.org]

On Tue, May 31, 2011 at 3:11 PM, Kenneth D. Merry <ken at freebsd.org> wrote:
> On Tue, May 31, 2011 at 15:02:37 -0700, mdf at FreeBSD.org wrote:
>> On Tue, May 31, 2011 at 2:46 PM, Kenneth D. Merry <ken at freebsd.org> wrote:
>> > On Tue, May 31, 2011 at 14:00:18 -0700, mdf at FreeBSD.org wrote:
>> >> On Tue, May 31, 2011 at 10:29 AM, Kenneth D. Merry <ken at freebsd.org> wrote:
>> >> > Author: ken
>> >> > Date: Tue May 31 17:29:58 2011
>> >> > New Revision: 222537
>> >> > URL: http://svn.freebsd.org/changeset/base/222537
>> >> >
>> >> > Log:
>> >> > ?Fix apparent garbage in the message buffer.
>> >> >
>> >> > ?While we have had a fix in place (options PRINTF_BUFR_SIZE=128) to fix
>> >> > ?scrambled console output, the message buffer and syslog were still getting
>> >> > ?log messages one character at a time. ?While all of the characters still
>> >> > ?made it into the log (courtesy of atomic operations), they were often
>> >> > ?interleaved when there were multiple threads writing to the buffer at the
>> >> > ?same time.
>> >>
>> >> This seems to panic my box with "lock "msgbuf" 0xfffffe0127ffffe0
>> >> already initialized".
>> >>
>> >> Unfortunately, though I booted with a fresh CURRENT this morning
>> >> successfully, both /boot/kernel and /boot/kernel.old give this panic.
>> >> To add insult to injury, when the kernel drops into the debugger, my
>> >> keyboard input no longer works so I can't get a stack, etc.
>> >
>> > Uh-oh!
>> >
>> >> So:
>> >>
>> >> 1) Is there anything else I can do to help debug this?
>> >> 2) how can I resurrect this box without a reinstall?
>> >>
>> >> I will try to repro on a virtual machine so I have a snapshot to come back to.
>> >
>> > My guess is that this is an issue with the message buffer reinitialization
>> > path. ?lock_init() (called by mtx_init()) has an assert to make sure that
>> > the lock is initialized, and that is just a flag check.
>> >
>> > Since the spin lock is part of the message buffer structure, if it is held
>> > over from a previous boot, the LO_INITIALIZED flag may still be set.
>> >
>> > Try power cycling the machine. ?If it is an issue with re-initialization,
>> > that should clear the memory and allow you to boot.
>>
>> Hmm, apparently my previous presses of the power button weren't long
>> enough.  I let it sit off for 20 seconds and it boots okay now.
>
> Okay, so it probably is the re-initialization code.  Can you try this patch
> and see if it survives a warm boot?  I also changed the initialization
> path, so we don't get tripped up by garbage left in memory.

This patch survived a warm reboot (shutdown -r now).

> Also, does the debugger work now that it has booted successfully?

The debugger (or rather, my keyboard in the debugger) works on a
successful boot.  I used sysctl debug.kdb.enter=1 to test it.

Thanks,
matthew