libpthread deadlock problem

Gergely, KOVACS (VBuster) gkovacs at virusbuster.hu
Wed Feb 9 05:23:06 PST 2005


Dear FreeBSD people,

I posted a libc deadlock problem here a few days ago. I ran compiled a
debug enabled libc and libpthread.

The software and the environment is the same:
A multi threaded program (an Anti-Virus for Sendmail with Milter
API compiled with gcc 3.4.2). The threads are created by Sendmail's
libmilter, and the number of threads is limited to 100
(MaxDaemonChildren is set to 100 in sendmail.cf)
The hardware environment is a dual Xeon 3Ghz, and the OS is FreeBSD 5.3
with SMP enabled kernel.
It ran with the default LIBPTHREAD_PROCESS_SCOPE=yes (M:N mode) pthread
setting.

The problem is that around in every 10th hour (100000 mail process) a
deadlock has been happened. According to gdb almost all threads are
waiting in malloc's, or free's mutex handler function:

#0  0x280b9cf3 in _thr_sched_switch_unlocked (curthread=0xcf3a400) at
pthread_md.h:225
#1  0x280b9c45 in _thr_sched_switch (curthread=0xcf3a400) at
/usr/src/lib/libpthread/thread/thr_kern.c:602
#2  0x280b652f in mutex_lock_common (curthread=0xcf3a400, m=0x280c5ba4,
abstime=0x0) at /usr/src/lib/libpthread/thread/thr_mutex.c:555
#3  0x280b765b in __pthread_mutex_lock (m=0x280c5ba4) at
/usr/src/lib/libpthread/thread/thr_mutex.c:796
#4  0x280afbfb in _spinlock (lck=0x28192f78) at
/usr/src/lib/libpthread/thread/thr_spinlock.c:93
#5  0x28122481 in pubrealloc (ptr=0x0, size=57, func=0x1 <Error reading
address 0x1: Bad address>) at /usr/src/lib/libc/stdlib/malloc.c:1092
#6  0x281226b8 in malloc (size=57) at
/usr/src/lib/libc/stdlib/malloc.c:1152

There is a thread which locked this (m=0x280c5ba4) mutex, and waiting
for an other:

#0  0x280b9cf3 in _thr_sched_switch_unlocked (curthread=0xccd0400) at
pthread_md.h:225
#1  0x280b9c45 in _thr_sched_switch (curthread=0xccd0400) at
/usr/src/lib/libpthread/thread/thr_kern.c:602
#2  0x280b652f in mutex_lock_common (curthread=0xccd0400, m=0x80a4784,
abstime=0x0) at /usr/src/lib/libpthread/thread/thr_mutex.c:555
#3  0x280b76d7 in _pthread_mutex_lock (m=0x80a4784) at
/usr/src/lib/libpthread/thread/thr_mutex.c:822
#4  0x280ac5ea in _sem_post (sem=0x8096334) at
/usr/src/lib/libpthread/thread/thr_sem.c:250
#5  0x280b1c8e in handle_signal (curthread=0x1, shi=0xbf39d7d8) at
/usr/src/lib/libpthread/thread/thr_sig.c:525
#6  0x280b1b0f in thr_sig_invoke_handler (curthread=0xccd0400, sig=14,
info=0x1, ucp=0xbf39d908) at
/usr/src/lib/libpthread/thread/thr_sig.c:489
#7  0x280b2691 in _thr_sig_rundown (curthread=0xccd0400, ucp=0xbf39d908,
psf=0x0) at /usr/src/lib/libpthread/thread/thr_sig.c:840
#8  0x280b290f in _thr_sig_check_pending (curthread=0xccd0400) at
/usr/src/lib/libpthread/thread/thr_sig.c:905
#9  0x280b9b1f in _kse_critical_leave (crit=0x1) at
/usr/src/lib/libpthread/thread/thr_kern.c:572
#10 0x280b9d6d in _thr_sched_switch_unlocked (curthread=0xccd0400) at
/usr/src/lib/libpthread/thread/thr_kern.c:671
#11 0x280b9c45 in _thr_sched_switch (curthread=0xccd0400) at
/usr/src/lib/libpthread/thread/thr_kern.c:602
#12 0x280b652f in mutex_lock_common (curthread=0xccd0400, m=0x280c5ba4,
abstime=0x0) at /usr/src/lib/libpthread/thread/thr_mutex.c:555
#13 0x280b765b in __pthread_mutex_lock (m=0x280c5ba4) at
/usr/src/lib/libpthread/thread/thr_mutex.c:796
#14 0x280afbfb in _spinlock (lck=0x28192f78) at
/usr/src/lib/libpthread/thread/thr_spinlock.c:93
#15 0x28122481 in pubrealloc (ptr=0x0, size=12, func=0x1 <Error reading
address 0x1: Bad address>) at /usr/src/lib/libc/stdlib/malloc.c:1092
#16 0x281226b8 in malloc (size=12) at
/usr/src/lib/libc/stdlib/malloc.c:1152
#17 0x08072514 in mi_rd_cmd ()
#18 0x0807108f in mi_engine ()
#19 0x08070f56 in mi_handle_session ()
#20 0x0807079e in mi_thread_handle_wrapper ()
#21 0x280aeb05 in thread_start (curthread=0xccd0400,
start_routine=0x8070790 <mi_thread_handle_wrapper>, arg=0xcc55d80)
    at /usr/src/lib/libpthread/thread/thr_create.c:342

There is no information who locked the mutex the above thread is waiting
for!

So could you help me how I can fix this?


Best Regards,
Gergely Kovacs



More information about the freebsd-bugs mailing list