Python and system vs process scope threads (was Re: pythonports broken (sem_destroy: Resource temporarily)

[Daniel Eischen]

On Fri, 6 Feb 2004, Julian Elischer wrote:

> 
> 
> On Sat, 7 Feb 2004, Hye-Shik Chang wrote:
> 
> > On Fri, Feb 06, 2004 at 06:11:57PM -0500, Daniel Eischen wrote:
> > > A brief synopsis:
> > > 
> > > The python port uses system scope threads, or at least the
> > > test_threaded_import.py test in /usr/local/lib/python2.3/test
> > > does.  Is there a reason python uses system scope threads
> > > instead of process scope threads?  It is more efficient
> > > and consumes less resources in FreeBSD if it were to use
> > > scope process threads.
> > 
> > Python uses system scope threads to make thread running run
> > like more human-expected. (The discussion was here:
> > http://groups.google.com/groups?threadm=slrn9ppgeb.mvb.bbrox%40bbland.bbrox.com&rnum=2)
> > With system scope, it runs:

The above thread seems to be Solaris related.  It looks like system
scope threads were used in order to get round-robin scheduling.  On
Solaris, you have to be in the real-time class in order to get FIFO
or RR scheduling, so the default is SCHED_OTHER (which is implementation
specific).

FreeBSD has for quite some time, even in libc_r, defaulted to round-robin
scheduling of threads.  The RR interval is I think 20 msec or so in 
libc_r, but it can be varied depending on what the system clock
rate (hz) is set to (I think the library checks hz and uses some
multiple of it).

For libpthread (nee libkse), it uses kern.clockrate also (which
is 10msec by default).

The test seems to be pretty simple and if it mattered in what order
the threads ran, then it would use synchronization primitives or
barriers to control scheduling of the threads.  Otherwise, it is
just appearances but doesn't really matter.

> > % python ~/testthread.py
> > thread 0 begin
> > thread 1 begin
> > thread 2 begin
> > thread 3 begin
> > thread 4 begin
> > thread 0 end
> > thread 1 end
> > thread 3 end
> > thread 2 end
> > thread 4 end
> > 
> > But with process scope, it runs:
> > 
> > % ./python ~/testthread.py
> > thread 0 begin
> > thread 0 end
> > thread 1 begin
> > thread 1 end
> > thread 2 begin
> > thread 2 end
> > thread 3 begin
> > thread 3 end
> > thread 4 begin
> > thread 4 end
> > 
> 
> what the threads do will effect when they are scheduled..
> 
> if you had 2 processors you ay also see:
> > thread 0 begin
> > thread 1 begin
> > thread 0 end
> > thread 1 end
> > thread 2 begin
> > thread 3 begin
> > thread 2 end
> > thread 3 end
> > thread 4 begin
> > thread 4 end
> 
> The link mentionnedabove is i think misguided..
> they are trying to make threads run concurrently, but..
> "why?"  If there is no dependency between the threads then
> what does it matter what order they are scheduled in, and if there IS a
> dependency, then teh threading package is supposed to take that into
> account, as one thread will block waiting for the other and the other
> will be allowed to run..

:-)

> Also even in Process-scope mode, threads will be round-robbinned
> a bit.. the test program probably doesn't wait long enough..
> (isn't that right Dan?)

Yep, see above.

> I don't see why the "concurrency" asked for in the mail shown above is
> important (other than looking pretty).
> I'd like to see a real-world example of why this is important.
> 
> 
> > 
> > Because our libc_r did system scope threads similar as process
> > scope, python threads have been scheduled like the latter one on
> > the previous FreeBSD releases. So even patching to use process scope
> > threads on the port will not hurt backward-compatibility.
> > 
> > Okay. I'll fix it on the port. Thanks for your investigation!
> > KSE rocks! ;)

Using scope system threads is certainly OK and allowed by POSIX.
We just didn't know why python defaulted to using them.  If it
is just appearances, then using scope process threads would
allow you to have many more threads (by default) and have
less impact on kernel resources.

-- 
Dan Eischen