Python with many threads

[Ott Köstner]

Michel Talon wrote:

> Nothing limits the number of concurrent threads. Personnally i have
> checked i can run Grub Next Generation Python Client with 600 threads
> without any problem.
> niobe% uname -a
> FreeBSD niobe.lpthe.jussieu.fr 7.0-STABLE FreeBSD 7.0-STABLE #0: Tue Jul
> 22 10:31:01 CEST 2008
> niobe% python
> Python 2.5.2 (r252:60911, Jul  5 2008, 13:44:44) 
>   
Still, something locks up in the system, when I increase the number 
Python threads with  http://grub.org/?q=en/node/204

With more than 50 threads I start to get random messages like:
Exception in thread 40:
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in 
__bootstrap_inner
    self.run()

The frequency of these messages increases with 70 threads. And with 200 
threads the system locks up totally -- almost no access from web. [I can 
still access the system and kill the process :)  ]

Clearly, there is some limit in the system. It is not memory, cpu or 
bandwidth. If the number of threads is also not limited, it must be 
something else. What might it be?


Network traffic is moderate:

IP traffic



> Recall that python threads are just native threads, but these threads
> are protected from trashing the python state by a single lock, the GIL,
> which is released when you do IO. This allows to effectively thread IO,
> but not python computation. The FreeBSD thread library has no particular
> limitations, you can run hundreds or thousands of threads without much
> problem, for example under Java.
>
>
>   



Regards,
O.K.

[sorry for HTML message in the list]

-- 
Testi oma Interneti kiirust / Test Your Internet speed:
http://speedtest.zzz.ee/