panic in propagate_priority w/ postgresql under heavy load

[Robert Watson]

On Mon, 19 Sep 2005, Koen Martens wrote:

> Without the debug stuff in the kernel, it crashed within 2 days, same 
> story: postgresql process, function propagate_priority. However, no dump 
> was written to disk :(
>
> Furthermore, i've been seeing the same crash (in propagate_priority) on 
> another box in mysql processes. Both servers seem to panic every 2-3 
> days. I have another server of the exact same hardware configuration, 
> but it is mainly idling most of the time. Haven't seen that one crash 
> yet.
>
> I am thinking now that it is a bug in the twa driver, so i'll have to 
> dig in to that. Furthermore, it seems to have to do with some sort of 
> concurrency issue or otherwise timing-sensitive issue, because slowing 
> the kernel down with debug code seems to avoid the panic. But, as i am 
> completely new to the freebsd kernel and don't even know what turnstiles 
> are, i imagine i will have a hard time. So if anyone can offer some 
> help, please :)
>
> Ok, thanks for your attention,

I can't speak to the problem with the core dumps, as it sounds like that 
is device/firmware related.  However, I probably can lend a hand in 
debugging the problems you're seeing.

First off, propagate_priority() is part of the priority propagation 
mechanism associated with mutexes, which are a locking primitive in the 
FreeBSD kernel.  Most panic in propagate_priority() are actually the 
result of a corrupted mutex, and when the mutex code goes to perform 
priority propagation, it trips over bad pointers and panics in some form 
of another.  Often, this means the actual panic or failure has not 
occurred in the thread that prints out the panic you see, but another 
panic.  So the first task on hitting a propagate_priority() panic is to 
identify the thread that actually had the problem.

Usually, I do this from DDB, rather than a core dump, because I find that 
DDB's tools for inspect running state are a little easier to use.  First, 
I identify what code called into the mutex call that resulted in 
propagate_priority() being called.  The reason to do this is that what you 
want to do next is use "ps" and "trace" to identify other 
processes/threads in the same code, and hence likely to have caused a 
problem with the mutex storage in memory.  Generally, you're looking for a 
panic in another thread, so once you identify a set of threads that might 
be to blame, you can trace them to find one that is in panic().  Usually, 
that thread will be in the RUN state, or on an SMP box, possibly running 
on another CPU.  If you're running 6.x, the thread that panicked was 
likely preempted as it had problems, perhaps due to an untimely interrupt.

If you want to do this by e-mail so we can lend a hand, you probably want 
to hook up a serial console so you can copy and paste the debugging 
session.  Compile DDB into the kernel (this should have no performance 
overhead), and when the system panics, you'll (ideally) get a db> prompt. 
The panic message and any related context (such as trap information) is 
useful.  I usually then use "show percpu" to see what CPU I'm running on, 
the thread that's running, etc.  I'll then use "trace" with no argument to 
see the stack of the thread.  If I'm trying to find another thread that 
may have been preempted, I'll use "ps" to show the running processes and 
threads, then "trace <pid>" to trace the main thread of processes that 
look interesting.  Generally, those in the RUN state, because the thread 
will be runnable.

If you're running on an SMP system, you may occasionally find that 
information to inspect the stacks of threads currently running on other 
processors may not be consistently in memory -- i.e., cached, the stack 
frame is partially written, or whatever.  There's a kernel option, 
KDB_STOP_NMI, which when combined with a sysctl, will cause the debugger 
to deliver an NMI IPI instead of a debug IPI, which may help kick those 
processors into the debugger if they are stuck in spin locks.  However, 
the chances are fairly good this isn't the case so you're probably fine 
without it.

Robert N M Watson