debugging frequent kernel panics on 8.2-RELEASE

[Attilio Rao]

2011/8/18 Andriy Gapon <avg at freebsd.org>:
> on 17/08/2011 23:21 Andriy Gapon said the following:
>>
>> It seems like everything starts with some kind of a race between
>> terminating
>> processes in a jail and termination of the jail itself.  This is where the
>> details are very thin so far.  What we see is that a process (http) is in
>> exit(2) syscall, in exit1() function actually, and past the place where
>> P_WEXIT
>> flag is set and even past the place where p_limit is freed and reset to
>> NULL.
>> At that place the thread calls prison_proc_free(), which calls
>> prison_deref().
>> Then, we see that in prison_deref() the thread gets a page fault because
>> of what
>> seems like a NULL pointer dereference.  That's just the start of the
>> problem and
>> its root cause.
>>
>> Then, trap_pfault() gets invoked and, because addresses close to NULL look
>> like
>> userspace addresses, vm_fault/vm_fault_hold gets called, which in its turn
>> goes
>> on to call vm_map_growstack.  First thing that vm_map_growstack does is a
>> call
>> to lim_cur(), but because p_limit is already NULL, that call results in a
>> NULL
>> pointer dereference and a page fault.  Goto the beginning of this
>> paragraph.
>>
>> So we get this recursion of sorts, which only ends when a stack is
>> exhausted and
>> a CPU generates a double-fault.
>
> BTW, does anyone has an idea why the thread in question would "disappear"
> from
> the kgdb's point of view?
>
> (kgdb) p cpuid_to_pcpu[2]->pc_curthread->td_tid
> $3 = 102057
> (kgdb) tid 102057
> invalid tid
>
> info threads also doesn't list the thread.
>
> Is it because the panic happened while the thread was somewhere in exit1()?
> is there an easy way to examine its stack in this case?

Yes it is likely it.

'tid' command should lookup the tid_to_thread() table (or similar
name) which returns NULL, which means the thread has past beyond the
point it was in the lookup table.

Attilio


-- 
Peace can only be achieved by understanding - A. Einstein