Too Much Context Switching?

[Reid Linnemann]

Written by alex at schnarff.com on 06/30/08 12:58>>
>>>> In 6.x. the default thread library is quite inefficient although it
>>>> can make use of multiple CPUs (again, providing the application is
>>>> giving them work to do).  For multi-threaded performance you will be
>>>> better off switching to the libthr library (see libmap.conf(5)) or
>>>> updating to 7.0 (where it is the default).  This isn't likely to be
>>>> the underlying issue if you are trying to debug a loss of
>>>> performance relative to the same configuration in the past though.
>>>
>>> Indeed Plone is written in python, and python has a "Big Giant Lock"
>>> inside which insures that only one thread can execute, in order to
>>> protect the python structures. This lock is only released under special
>>> circumstances, such as doing IO. Hence it is necessary to run several
>>> instances of python programs and do synchronization work, if one wants
>>> to make use of several CPUs, or use python threads, and immediately
>>> make some IOs, or similar techniques. It may be that using Jython, if
>>> possible, yields better threading behavior. When doing some work
>>> according to these ideas, i had found quite severe contention, and this
>>> was not cured when switching native threading libraries (libksd, libthr,
>>> etc.). The problem is really inside python.
>>
>> Yep, it could be that -- what confuses me though is that it is claimed
>> that performance suddenly regressed.  If so then this cannot be the
>> underlying cause.
>>
> 
> It's actually been a long, slow, steady degradation of performance as
> best I can tell, that's recently just reached proportions that are so
> ridiculous that it's gone from "this sucks but I can deal" to "this is
> completely unusable." The system has been slow from the start, just not
> this slow. I guess I'll need to investigate this...and while I know that
> Python is somewhat off-topic, if anyone here has any suggestions on
> where to start, they'd be much appreciated. :-)
> 
> Alex

As far as degradations-over-time are concerned, don't overlook your
ZODB. If you don't pack it regularly and it grows to some ridiculous
size you can be in for a world of hurt.