python and HUGE_STACK_SIZE

[Andrew MacIntyre]

Doug Barton wrote:
> On 03/26/10 14:03, Steven Kreuzer wrote:
>> So, it seems like most of the time python scripts will work with HUGE_STACK_SIZE
>> turned off, but every once and a while some scripts will fail in non obvious ways
>> that could leave a person scratching their head for weeks trying to get to the bottom of it
>>
>> To me, it seems like the best behavior would be to default to compiling with that set. I'll create
>> a patch over the weekend and open a PR
> 
>>From the discussion (not speaking from experience or python knowledge)
> it seems like an OPTION is the way to go, with the open question being
> defaults to on or defaults to off.
> 
> What is the impact of HUGE_STACK_SIZE when it's compiled in, and how
> will it affect those running python stuff who don't actually need it?
> 
> If it turns out that only a few ports need it and the impact is
> undesirable those ports that do need it could be adapted to test for it
> somehow and suggest that the user re-install python with the option.

HUGE_STACK_SIZE actually refers to the stack size used for each thread 
other than the primary thread.  As I understand it, these stacks are 
fully committed while the thread exists, which is different to the 
handling of the primary thread's stack (where only the pages used are 
committed).  On 32 bit systems, careless use of large thread stacks can 
cause non-trivial address space wastage.

People don't see the same issues on Linux as it has a much larger 
default thread stack size and I'm lead to believe that only the pages in 
use in the thread stacks are committed.

Since Python 2.5, there has been a way to change the thread stack size 
at runtime: threading.stack_size().

Ports patches using these calls for packages likely to be affected (such 
as Zope) could make this easier for a lot of people, if support for them 
can't be encouraged upstream.

-- 
-------------------------------------------------------------------------
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au  (pref) | Snail: PO Box 370
        andymac at pcug.org.au             (alt) |        Belconnen ACT 2616
Web:    http://www.andymac.org/               |        Australia