threads/123062: C++ exception handling can loop during stacking unwinding in multithreaded programs

[Andy Newman]

>Number:         123062
>Category:       threads
>Synopsis:       C++ exception handling can loop during stacking unwinding in multithreaded programs
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-threads
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Apr 25 01:10:02 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator:     Andy Newman
>Release:        7-STABLE
>Organization:
>Environment:
FreeBSD juju.bsn 7.0-STABLE FreeBSD 7.0-STABLE #22: Fri Apr 18 20:18:44 EST 2008 root at juju.local:/usr/obj/usr/src/sys/JUJU  i386


>Description:
The gcc runtime's _Unwind_Find_FDE function, invoked during exception handling's stack unwinding, is not safe to execute from within multiple threads.  FreeBSD's dl_iterate_phdr() however permits multiple threads to pass through it though.  The result is surprisingly reliable infinite looping of one or more threads if they just happen to be unwinding at the same time.

A bug report to gcc (http://gcc.gnu.org/bugzilla/show_bug.cgi?id=36030), with a fix to _Unwind_Find_FDE to protect its cache manipulations, has the response that glibc's dl_iterate_phdr doesn't  let multiple threads unwind at the same time.  To that end here's a patch to ld-elf.so that adds a lock around the dl_iterate_pdr call (well, within it) that has it operate in the same way as glibc's and avoid the looping problem.

>How-To-Repeat:
Reproduction is via the small program attached to the gcc bug report.  As described there, the program creates some number of threads, each of which immediately throws an exception. The exceptions are caught and ignored in the thread.  The threads are then joined and the program exits.

Failure does not occur every time due to the usual race condition issues however by running the program in a shell loop it will eventually lock up attempting to join the threads.  Some number of threads will be looping consuming 100% CPU.

This has been tested on multiprocessor machines (two and four way) will reliable behavior.


>Fix:
Apply the attached patch and re-run tests.  No failures have been observed over hundreds of thousands of runs.

The patch adds another lock to ld-elf.so used solely to stop multiple threads running dl_iterate_pdr() at the same time.  At entry the lock is obtained (as a write lock), the iteration performed and then the lock is released.

Note that simply upgrading the existing read lock (rtld_bind_lock) to a write lock does not work and results in deadlock presumably due to other rtld functions being called during the stack unwinding which want to obtain the "bind" lock.



Patch attached with submission follows:

Index: rtld.c
===================================================================
RCS file: /home/ncvs/src/libexec/rtld-elf/rtld.c,v
retrieving revision 1.124
diff -u -r1.124 rtld.c
--- rtld.c	17 May 2007 18:00:27 -0000	1.124
+++ rtld.c	25 Apr 2008 00:34:54 -0000
@@ -2100,7 +2100,7 @@
     const Obj_Entry *obj;
     int error, lockstate;
 
-    lockstate = rlock_acquire(rtld_bind_lock);
+    lockstate = wlock_acquire(rtld_phdr_lock);
 
     error = 0;
 
@@ -2119,7 +2119,7 @@
 		break;
 
     }
-    rlock_release(rtld_bind_lock, lockstate);
+    wlock_release(rtld_phdr_lock, lockstate);
 
     return (error);
 }
Index: rtld_lock.c
===================================================================
RCS file: /home/ncvs/src/libexec/rtld-elf/rtld_lock.c,v
retrieving revision 1.4
diff -u -r1.4 rtld_lock.c
--- rtld_lock.c	3 Apr 2007 18:28:13 -0000	1.4
+++ rtld_lock.c	24 Apr 2008 13:55:04 -0000
@@ -171,7 +171,7 @@
 	lockinfo.thread_clr_flag(mask);
 }
 
-#define	RTLD_LOCK_CNT	2
+#define	RTLD_LOCK_CNT	3
 struct rtld_lock {
 	void	*handle;
 	int	 mask;
@@ -179,6 +179,7 @@
 
 rtld_lock_t	rtld_bind_lock = &rtld_locks[0];
 rtld_lock_t	rtld_libc_lock = &rtld_locks[1];
+rtld_lock_t	rtld_phdr_lock = &rtld_locks[2];
 
 int
 rlock_acquire(rtld_lock_t lock)
Index: rtld_lock.h
===================================================================
RCS file: /home/ncvs/src/libexec/rtld-elf/rtld_lock.h,v
retrieving revision 1.2
diff -u -r1.2 rtld_lock.h
--- rtld_lock.h	19 Jun 2003 02:39:37 -0000	1.2
+++ rtld_lock.h	24 Apr 2008 13:55:16 -0000
@@ -52,6 +52,7 @@
 
 extern rtld_lock_t	rtld_bind_lock;
 extern rtld_lock_t	rtld_libc_lock;
+extern rtld_lock_t	rtld_phdr_lock;
 
 int	rlock_acquire(rtld_lock_t);
 int 	wlock_acquire(rtld_lock_t);


>Release-Note:
>Audit-Trail:
>Unformatted: