Passenger hangs on live and SEGV on tests possible threading / kernel bug?

[John Baldwin]

On Thursday 17 December 2009 6:12:07 am Steven Hartland wrote:
> We're having an issue with Passenger on FreeBSD where it will hang
> and stop processing any more requests the details are attach to
> the following bug report:
> http://code.google.com/p/phusion-passenger/issues/detail?id=318#c14
> 
> In addition the test suite crashes in what seems to be a very
> basic test, which I'm at a loss with.
> http://code.google.com/p/phusion-passenger/issues/detail?id=441
> 
> I'm thinking this may be a bugs in the FreeBSD either kernel or
> thread library as the crashes don't make any sense from the
> application side.
> 
> Any advise on debugging or feedback on the stack traces would
> be much appreciated.

For the hang it seems you have a thread waiting in a blocking read(), a thread 
waiting in a blocking accept(), and lots of threads creating condition 
variables.  However, the pthread_cond_init() in libpthread (libthr on FreeBSD) 
doesn't call pthread_cleanup_push(), so your stack trace doesn't make sense to 
me.  However, that may be gdb getting confused.  The pthread_cleanup_push() 
frame may be cond_init().  However, it doesn't call umtx_op() (the 
_thr_umutex_init() call it makes just initializes the structure, it doesn't 
make a _umtx_op() system call).  You might try posting on threads@ to try to 
get more info on this, but your pthread_cond_init() stack traces don't really 
make sense.  Can you rebuild libc and libthr with debug symbols?

For example:

# cd /usr/src/lib/libc
# make clean 
# make DEBUG_FLAGS=-g
# make DEBUG_FLAGS=-g install

However, if you are hanging in read(), that usually means you have a socket 
that just doesn't have data.  That might be an application bug of some sort.

The segv trace doesn't include the first part of GDB messages which show which 
thread actually had a seg fault.  It looks like it was the thread that was 
throwing an exception.  However, nanosleep() doesn't throw exceptions, so that 
stack trace doesn't really make sense either.  Perhaps that stack is hosed by 
the exception handling code?

-- 
John Baldwin