libkse and SMP (was Re: USB bulk read & pthreads)

[Daniel Eischen]

On Thu, 22 May 2003, Terry Lambert wrote:

> Dan Nelson wrote:
> > In the last episode (May 22), Terry Lambert said:
> > > Make sure you use SCHED_4BSD, rather than SCHED_ULE, if you are using
> > > one of the kernel threads libraries, for now.
> > >
> > > You really should read the -current archives before attempting any of
> > > this, if you don't follow -current closely enough to have caught
> > > Jeff's message on this, or which kernel threading libraries are
> > > available.
> > 
> > I do read -current, and I'm definitely not going anywhere near
> > SCHED_ULE.  I just tested both libraries with today's kernel and
> > libraries, and was able to get a hard lockup with both libthr and
> > libkse.  Mysql seems to run okay.  Starting a threaded pike process
> > seems to be the killer.  Unfortunately, pike's a pretty large app so
> > it's not easy to get a stripped-down testcase.
> 
> The philosophies behing libthr and libkse are different.  For
> libthr, you basically have a FreeBSD version of Linux threads;
> for libkse, there are some issues you have to deal with.
> 
> The first of these issues is that you have to add Daniel's patch
> to the libkse scheduler code.  This is necessary because there is
> a lot of threads code that is not completely compliant with the
> POSIX standard: it makes assumptions which POSIX does not permit,
> about what thread gets scheduled to run after an involuntary
> context switch.  The original Netscape (e.g. 4.7) has this same
> assumption, and Java interfaces that use image maps lock it up if
> you move the mouse over the map while the GIF is loading.  If you
> wait for it to load, there's no problem.  Mozilla has similar
> assumptions.

This is just because rtld-elf is not thread safe (or written
so that locks are not needed as you claim).

> The second of these is that the libkse model is M:N, with N being
> defaulted to 1.  If you want more kernel threads, you have to ask

N is defaulted to the number of CPUs that you have.  Libkse
will create as many KSEs as there are CPUs (or whatever
kern.threads.virtual_cpu is set to); these will be KSEs
that run scope process threads.  Scope system threads
get their own KSE/KSEG pair without regard to number of
CPUs or kern.threads.virtual_cpu.

Trying to exceed the number of CPUs with kern.threads.virtual_cpu
will not work unless you also set kern.threads.debug=1.  You
really shouldn't need to do this, but we use it for testing
and debugging.

> for them (and you aren't doing that).  In general, a lot of code
> is going to assume for M:N that M==N; to get this, you will need
> to create threads with an attribute other than the default of
> "NULL", with the scope set to PTHREAD_SCOPE_SYSTEM.  You should
> also look at the actual scheduler code in libkse to see if there
> are any other requirements for creating KSEG's for KSE's for
> user space threads (I haven't looked at it for several weeks now,
> so I don't know if anything has changed, unless it was sent to
> one of the mailing lists).
> 
> With both of these out of the way, libkse should work for you.
> 
> The libthr may not work for you due to scheduling order, or it
> may not work for you because signals are not masked per process,
> or it may not work if you have threads that depend on the
> PTHREAD_SCOPE_PROCESS attribute (kernel threads like the Linux
> model are inherently incapable of supporting this scope).
> 
> Basically, you'll have to play around a bit with it.
> 
> -- Terry
> 

-- 
Dan Eischen