libpthread patch

[Daniel Eischen]

There's an updated patch file available at (a slightly different place):

    http://people.freebsd.org/~deischen/kse/libpthread.diffs

There's also an html'ized log of the ACE tests:

    http://people.freebsd.org/~deischen/kse/ace_build_logs/index.html

The only real problems seem to be with the ACE tests:

    Cached_Conn_Test
    Process_Manager_Test

And I think these have something to do with wait() or waitpid()
not working correctly.  David, do you know of any problems in
this area?  It seems that sometimes waitpid() is returning 0
and the next time it is called it returns the process id.
I wonder if it is being interrupted by a signal (either the
kernel doing it or the UTS by use of kse_thr_interrupt)?

Other than that, things are looking pretty good.  The code
needs some cleanup; I'll try to do that and see if I can
get rid of the %gs munging as well.

On Mon, 14 Apr 2003, David Xu wrote:
> 
> ----- Original Message ----- 
> From: "Daniel Eischen" <eischen at pcnet1.pcnet.com>
> To: "David Xu" <davidxu at freebsd.org>
> Cc: <freebsd-threads at freebsd.org>
> Sent: Monday, April 14, 2003 12:55 PM
> Subject: Re: libpthread patch
> 
> 
> > > I think the whole scheduler lock should be reworked
> > > to allow state transition is in atomic, my change is
> > > not SMP safe, only works on UP, because kse_critical_enter
> > > is only works for UP system. If we fixed this scheduler lock
> > > problem, I think the libpthread will be stable enough.
> > 
> > I thought that we might want to make changing the state
> > and switching to the scheduler all under the same lock,
> > but decided not to go that way.  I was concerned that by
> > requiring the scheduler lock to be held while atomically
> > changing the thread state and switching to the scheduler
> > might introduce weird locking order and wanted to avoid
> > that.  For instance, inherited priority adjustments and
> > condition signal broadcasts.  We should look at it again
> > though, because it may make sense in most cases.
> > 
> 
> However, I found that because there are multiple schedule
> queues, is it possible that there is deadlock between those
> queues? I mean threads are inter-locking each other's schedule
> queue.

Right, but this is OK as long as you don't hold your own
scheduler lock when you are trying to lock another one.

> > I'll try to get a new patch set out after I fix
> > pthread_join().  I should have posted my earlier version
> > but I thought I could fix join quickly.  It turns out
> > it's a bit more tricky than I thought.
> > 
> > One question about kse_create().  I was thinking about
> > implementing pthread_setconcurrency().  We need to return
> > an error if the requested concurrency level cannot be
> > achieved.  If we do return an error, though, we should
> > not end up creating _any_ additional KSEs.  If someone
> > does:
> > 
> > ret = pthread_setconcurrency(5);
> > 
> > How do we know how many KSEs (within the same KSEG) we can
> > create?  And if we use ncpu or some other knob, it doesn't
> > automatically mean we _will_ get the number of KSEs requested.
> > We can currently only make one at a time, so we may end up
> > with some additional KSEs but less than requested -- but we
> > still have to return an error to the caller.
> > 
> 
> there is a sysctl kern.threads.virtual_cpu, it tells application
> that max kses can be created in same ksegroup. it is independent
> from physical CPU, although it may equal number of physical CPU.
> the sysctl can be changed by root, but does not have effect,
> unless kern.threads.debug = 1, I might change it to not masked by
> kern.threads.debug.
> current kse_create prevents userland from creating kses (upcall)
> more than kern.threads.virtual_cpu, it will return EPROCLIM
> if userland is trying to do.
> I think kern.threads.virtual_cpu is what you want.

Right, but there is no easy way for the UTS to ask for 4
additional KSEs and be certain that it will get 4.  It can
check the sysctl knob, but the kernel may still fail to
create additional KSEs (resource starvation?).  Since
the UTS can only create them one at a time, it won't
_really_ know if it can create 4 until after it creates
the 4th one.  If there is an error creating KSE number
2, 3, or 4, it can still return an error to the caller
(from pthread_setconcurrency()), but there is an unexpected
side-effect of having _some_ additional KSEs created.

This isn't a big deal for now, but to be really correct,
I think we might have to destroy any additional KSEs if
we didn't get the full amount requested.

> Notic about libpthread patch:
> libpthread_init does not link _kse_initial into it's kseg, why?
> I have linked it. and I also fixed scheduler lock in _thr_alloc()
> and _thr_free().

I'll take a look at those.

> in pthread_cancel, I think it should use thr_add_ref not
> thr_find, because it is possible before pthread_cancel processes
> that thread, the thread may already be exited.

I removed thr_find() and changed it to thr_ref_add().  There is
also a new argument to thr_ref_add(, , /* include dead */ int)
so you can specify whether or not you want to include dead threads
in your search.  pthread_join() wants to include dead threads,
whereas most of the others don't.

-- 
Dan Eischen