python segfaulted, all threads in pthread_testcancel

[Jay Moorthi]

Hi Folks,

I've been seeing python 2.4.2 processes crashing every now and then,
running a handful of different scripts that use the threading module.  I
can't reliably reproduce the crash, and I have not yet noticed a pattern
to the failures other than their similar stack traces.  Every thread
shows up in pthread_testcancel.

The python binary I'm dealing with is stripped, so I don't have lots of
debugging info, but I get the following when I open the latest example
core in gdb:

(gdb) info threads
  5 process 100191  0x881351af in pthread_testcancel () from
/usr/lib/libpthread.so.2
  4 process 100227  0x881351af in pthread_testcancel () from
/usr/lib/libpthread.so.2
  3 process 100140  0x881351af in pthread_testcancel () from
/usr/lib/libpthread.so.2
  2 process 100187  0x8813520f in pthread_testcancel () from
/usr/lib/libpthread.so.2
* 1 process 100126  0x881351ef in pthread_testcancel () from
/usr/lib/libpthread.so.2
(gdb) thread apply all bt

Thread 5 (process 100191):
#0  0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2
#1  0x8812dbd0 in pthread_mutexattr_init () from
/usr/lib/libpthread.so.2
#2  0x00000000 in ?? ()

Thread 4 (process 100227):
#0  0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2
#1  0x8812e517 in pthread_mutexattr_init () from
/usr/lib/libpthread.so.2
#2  0x08521000 in ?? ()

Thread 3 (process 100140):
#0  0x881351af in pthread_testcancel () from /usr/lib/libpthread.so.2
#1  0x8812e517 in pthread_mutexattr_init () from
/usr/lib/libpthread.so.2
#2  0x08521000 in ?? ()

Thread 2 (process 100187):
#0  0x8813520f in pthread_testcancel () from /usr/lib/libpthread.so.2
#1  0x080bc702 in PyThread_release_lock ()
#2  0x0809b8e7 in PyEval_EvalFrame ()
#3  0x080a0100 in PyEval_EvalCodeEx ()
#4  0x080d54f6 in PyFunction_SetClosure ()
#5  0x0805a448 in PyObject_Call ()
#6  0x0805fce6 in PyMethod_New ()
#7  0x0805a448 in PyObject_Call ()
#8  0x0809a767 in PyEval_CallObjectWithKeywords ()
#9  0x0808339d in _PyObject_SlotCompare () #10 0x0807e22c in
PyType_GenericNew ()
#11 0x080d3aa8 in PyGen_New ()
#12 0x080b6e52 in PyThreadState_Clear ()
#13 0x080b6e85 in PyInterpreterState_Clear ()
#14 0x080b9436 in Py_Finalize ()
#15 0x08055578 in Py_Main ()
#16 0x0805503d in main ()

Thread 1 (process 100126):
#0  0x881351ef in pthread_testcancel () from /usr/lib/libpthread.so.2
#1  0x8812e2cd in pthread_mutexattr_init () from
/usr/lib/libpthread.so.2
#2  0x8812e3ae in pthread_mutexattr_init () from
/usr/lib/libpthread.so.2
#3  0x08521000 in ?? ()
(gdb)

Is pthread_testcancel here a red herring?  The ??s at the outermost
frames make me think that gdb is getting the stack unwind slightly wrong
and landing in pthread code by mistake, but thread 2's stack seems
plausible.

Is there anything I can do to determine whether these are sane
backtraces?

The system is running 6.0-STABLE:

moorthi at wbrs2-app2$ uname -a
FreeBSD wbrs2-app2.soma.ironport.com 6.0-STABLE FreeBSD 6.0-STABLE #0:
Sat May 27 01:33:34 UTC 2006
root at wbrs2-app2.soma.ironport.com:/usr/src/sys/i386/compile/MESSAGING_GA
TEWAY.i386_INSTALL  i386
moorthi at wbrs2-app2$

moorthi at wbrs2-app2$ ldd /usr/local/bin/python
/usr/local/bin/python:
        libpthread.so.2 => /usr/lib/libpthread.so.2 (0x88118000)
        libutil.so.5 => /lib/libutil.so.5 (0x8813d000)
        libstdc++.so.5 => /usr/lib/libstdc++.so.5 (0x88149000)
        libm.so.4 => /lib/libm.so.4 (0x88213000)
        libc.so.6 => /lib/libc.so.6 (0x88229000)

moorthi at wbrs2-app2$ more /etc/libmap.conf [mysqld]
libpthread.so.2 libthr.so.2
libpthread.so libthr.so

Another piece of information is it appears that these python processes
do make significant progress in script executiong before they segfault.
In general, they are all connected to one of many MySQL database servers
using MySQLdb.

Any advice you may have will be useful, especially suggestions on where
I should look for more information.

Thanks,

Jay