Complete hang on 9.0-RELEASE

[Arnaud Lacombe]

Hi,

On Wed, Feb 29, 2012 at 2:31 PM, Arnaud Lacombe <lacombar at gmail.com> wrote:
> Hi,
>
> On Wed, Feb 29, 2012 at 2:22 PM, Attilio Rao <attilio at freebsd.org> wrote:
>> 2012/2/29, Arnaud Lacombe <lacombar at gmail.com>:
>>> Hi,
>>>
>>> On Wed, Feb 29, 2012 at 1:44 PM, Attilio Rao <attilio at freebsd.org> wrote:
>>>> 2012/2/29, Arnaud Lacombe <lacombar at gmail.com>:
>>>>> Hi,
>>>>>
>>>>> On Wed, Feb 29, 2012 at 12:59 PM, Arnaud Lacombe <lacombar at gmail.com>
>>>>> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On Mon, Feb 27, 2012 at 12:48 PM, Arnaud Lacombe <lacombar at gmail.com>
>>>>>> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Mon, Feb 27, 2012 at 10:36 AM, Attilio Rao <attilio at freebsd.org>
>>>>>>> wrote:
>>>>>>>> 2012/2/27, Arnaud Lacombe <lacombar at gmail.com>:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On Tue, Feb 14, 2012 at 11:41 AM, Arnaud Lacombe <lacombar at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>> Hi folks,
>>>>>>>>>>
>>>>>>>>>> For the records, I was running some tests yesterday on top of a
>>>>>>>>>> 9.0-RELEASE, amd64, kernel when the box hanged. At the time of the
>>>>>>>>>> hang, the box was running a process with about 2800 threads with
>>>>>>>>>> heavy
>>>>>>>>>> IPC between 1400 writers and 1400 readers. The box was in single
>>>>>>>>>> user
>>>>>>>>>> mode (/bin/sh coming from FreeBSD 7.4-STABLE). Here is the beginning
>>>>>>>>>> of the dmesg:
>>>>>>>>>>
>>>>>>>>> This happened a second time, now with FreeBSD 8.2-RELEASE. Complete
>>>>>>>>> machine hang. The machine was running about 4000 threads in a single
>>>>>>>>> process, all the other condition are the same.
>>>>>>>>
>>>>>>>> Arnaud,
>>>>>>>> can you please break in your kernel via KDB, collect the following
>>>>>>>> informations from the DDB prompt:
>>>>>>>> - ps
>>>>>>>> - alltrace
>>>>>>>> - show allpcpu
>>>>>>>> - possibly get a coredump with 'call doadump'
>>>>>>>>
>>>>>>> Will do, but I'll need to rebuild a kernel to include DDB.
>>>>>>>
>>>>>>>> and in the end provide all those along with kernel binary and possibly
>>>>>>>> sources somewhere?
>>>>>>>>
>>>>>>> I'll be testing a bare `release/8.2.0' with the following patch:
>>>>>>>
>>>>>>> diff --git a/sys/amd64/conf/GENERIC b/sys/amd64/conf/GENERIC
>>>>>>> index c3e0095..7bd997f 100644
>>>>>>> --- a/sys/amd64/conf/GENERIC
>>>>>>> +++ b/sys/amd64/conf/GENERIC
>>>>>>> @@ -79,6 +79,10 @@ options      INCLUDE_CONFIG_FILE     # Include this
>>>>>>> file in kernel
>>>>>>>
>>>>>>>  options        KDB           # Kernel debugger related code
>>>>>>>  options        KDB_TRACE     # Print a stack trace for a panic
>>>>>>> +options        DDB
>>>>>>> +options        BREAK_TO_DEBUGGER
>>>>>>> +options        ALT_BREAK_TO_DEBUGGER
>>>>>>>
>>>>>>>  # Make an SMP-capable kernel by default
>>>>>>>  options        SMP           # Symmetric MultiProcessor Kernel
>>>>>>>
>>>>>> ok, it happened again after 2 days, the process was running about 3200
>>>>>> threads. I'm trying to break into DDB and let you know, I'm not that
>>>>>> successful for now...
>>>>>>
>>>>> No luck. None of BREAK or ALT_BREAK are responding. I will not touch
>>>>> the system in the next few hours if you want me to test something on
>>>>> it. In the event of 8.2-RELEASE or 9.0-RELEASE are  not meant to work
>>>>> reliably on top of a 7.4-RELEASE userland, I will re-setup the test to
>>>>> occurs on a clean 9.0-RELEASE system and re-try.
>>>>
>>>> We allow to break KBI when new releases happens, thus this may cause a
>>>> breakage for you, even if a deadlock is really not something you want.
>>>>
>>>> Can you try enabling SW_WATCHDOG, DEADLKRES and possibly arm your ichwd?
>>>> if the breakage involves clocks or interrupt sources there are still
>>>> chances they will be able to catch it though.
>>>>
>>>> However, it doesn't seem you are setup with a proper serial console?
>>> The serial console is working definitively fine. I can break into DDB
>>> at will when the test is running. I did not test with ALT_BREAK
>>> per-se, but BREAK does work.
>>
>> So if you try to break in DDB via serial break it doesn't work?
>> That is definitively very bad...
>>
> just to be sure, I rebooted the system and I could break into DDB at
> the first attempt with ALT_BREAK, BREAK was a bit more reluctant but
> worked too. So yes, this does not taste good :/
>
>> Can you try with the options I mentioned earlier and see if something changes?
>>
> will do, but I will first attempt to reproduce this on 9.0-RELEASE.
>
9.0-RELEASE (kernel + userland) hanged today while running 2000
threads. Next step is to reproduce it with a watchdog+textdump enabled
kernel.

 - Arnaud