threaded, forked, rethreaded processes will deadlock

[Brian Fundakowski Feldman]

On Fri, Jan 09, 2009 at 07:42:32PM +0200, Kostik Belousov wrote:
> On Fri, Jan 09, 2009 at 11:34:26AM -0500, Brian Fundakowski Feldman wrote:
> > On Thu, Jan 08, 2009 at 11:09:16PM -0800, Julian Elischer wrote:
> > > Brian Fundakowski Feldman wrote:
> > >> On Thu, Jan 08, 2009 at 10:44:20PM -0500, Daniel Eischen wrote:
> > >>> On Thu, 8 Jan 2009, Brian Fundakowski Feldman wrote:
> > >>> 
> > >>>> It appears that the post-fork hooks for malloc(3) are somewhat broken such that
> > >>>> when a threaded program forks, and then its child attempts to go threaded, it
> > >>>> deadlocks because it already appears to have locks held.  I am not familiar
> > >>>> enough with the current libthr/libc/rtld-elf interaction that I've been able
> > >>>> to fix it myself, unfortunately.
> > >>> There's really nothing to fix - according to POSIX you are only
> > >>> allowed to call async-signal-safe functions in the child forked
> > >>> from a threaded process.  If you are trying to do anything other
> > >>> than that, it may or may not work on FreeBSD, but it is not
> > >>> guaranteed and is not portable.
> > >>> 
> > >>> The rationale is that what is the point of forking and creating
> > >>> more threads, when you can just as easily create more threads in
> > >>> the parent without forking?  The only reason to fork from a threaded
> > >>> process is to call one of the exec() functions.
> > >> 
> > >> Well, it worked until the last major set of changes to malloc.  For me, the point
> > >> was that I was able to have transparent background worker threads in any program
> > >> regardless of its architecture, using the standard pthread fork hooks.  Could you
> > >> point me to the POSIX section covering fork and threads?  If it's really not
> > >> supposed to work then that's fine, but there's an awful lot of code there dedicated
> > >> to support going threaded again after a fork.
> > >> 
> > > 
> > > Practically, you can't go threaded again after a fork
> > > (by which I mean creating new threads or use things
> > > like mutexes etc.) in any posix system I know of.
> > > 
> > > It would require that:
> > >  The forking thread stop until:
> > >   Every other thread has released every resource it owns
> > >   and reports itself to be in a "safe quiescent state",
> > >   or at least report every resource it owns, especially
> > >   locks,
> > >  and
> > >  After the fork:
> > >   The child, post fork, to take ownership of all
> > >   of them, and free them.
> > > 
> > > You might be able to do that in a simple
> > > threaded program, but consider then that the libraries may have
> > > threads running in them of which you are unaware, and that
> > > some of the resources may interract with resources owned by the
> > > forking thread.
> > > 
> > > Add to this that there may be a signal thrown into this mix as well
> > > 
> > > (signals are the bane of thread developement)
> > 
> > Well, I wouldn't mind showing all of you what I can of what I had been doing
> > with the background threads -- it works pretty well modulo this particular
> > malloc lock bug.  Due to it being inappropriate to share library resources
> > between a child and parent for an open socket connection, I always considered
> > the only "safe" behavior to be going single-threaded for the duration of the fork
> > processes in both the parent and child, and the pthread_atfork(3) hooks have been
> > sufficient to do so.
> > 
> In fact, try recent HEAD, it contains the fixed for deadlocks caused by
> fork calls in mt processes. Also, see r185456, that may be directly
> relevant to your problem.
> 
> The MFC to the stable/7 is doable, but depends on the MFC of some stuff
> committed by David Xu, that would take me some time to look into.

Ah, thank you for the pointer.  Seems this is an opportunity for me to catch up with
our switch to SVN, then.  I'll take a further look at this this weekend.  For reference,
these platforms passed the particular regression test I posted:
Linux 2.6.18-92.1.17.el5
SunOS 5.10 Generic_137138-09
Darwin macintosh.green.homeunix.org 9.6.0 Darwin Kernel Version 9.6.0: Mon Nov 24 17:37:00 PST 2008;root:xnu-1228.9.59~1/RELEASE_I386 i386
FreeBSD 7.0-RELEASE
-- 
Brian Fundakowski Feldman                           \'[ FreeBSD ]''''''''''\
  <> green at FreeBSD.org                               \  The Power to Serve! \
 Opinions expressed are my own.                       \,,,,,,,,,,,,,,,,,,,,,,\