ace/freebsd: THR_NEW_LWP problem with libpthread/pthread_setconcurrency

[Daniel Eischen]

On Wed, 30 Aug 2006, Andriy Gapon wrote:

>
> ACE VERSION:
> 5.5.1 (from FreeBSD port: ace-5.5.1)
>
> HOST MACHINE and OPERATING SYSTEM:
> FreeBSD 6.1-RELEASE-p2 i386 (uniprocessor)
>
> COMPILER NAME AND VERSION (AND PATCHLEVEL):
> gcc version 3.4.4 [FreeBSD] 20050518 (system compiler)
> libpthread is used as threading library
>
> AREA/CLASS/EXAMPLE AFFECTED:
> ACE threads
>
> DOES THE PROBLEM AFFECT:
> EXECUTION - this is a run-time problem
>
> SYNOPSIS:
> Can not create threads with THR_NEW_LWP flag.
>
> DESCRIPTION:
> ACE_Thread_Manager::spawn(..., THR_NEW_LWP) fails with EAGAIN. It seems
> that this happens because when creating such threads ACE calls
> pthread_setconcurrency with incrementally increasing concurrency levels.
> pthread_setconcurrency of libpthread in turn seems to call kse_create()
> which fails with EPROCLIM. As I understand this happens because kernel
> does not allow to have more KSE-s per process as there are processors.

pthread_setconcurrency() should return EAGAIN, not EPROCLIM.  kse_create()
may set errno to EPROCLIM, but I don't see how the code in libpthread
can return anything other than EAGAIN if kse_create() fails.

POSIX states that pthread_setconcurrency() can return EAGAIN, so I
think ACE is at fault if it is not handling that properly.  I
would also think that THR_NEW_LWP should not increment the
concurrency, but create a new system scope thread since that
seems more in the spirit of "spawning a new thread with a new
LWP".

> This problem is quite noticeable with ACE because THR_NEW_LWP is present
> in default flags in many places.
>
> REPEAT BY:
> Running any ACE-based program that spawns threads with THR_NEW_LWP and
> linked to libpthead.
>
> SAMPLE FIX/WORKAROUND:
> Straightforward workaround is to explicitly pass thread flags everywhere
> and not use THR_NEW_LWP. Maybe this could be patched in ACE sources as a
> part of FreeBSD port patch step.
> IMHO, better fixes would be:
> 1. make ACE_Thread_Manager::spawn() more robust with respect to
> pthread_setconcurrency() failing with EAGAIN.
> 2. make FreeBSD libpthread dumb-happy in pthread_setconcurrency(), i.e
> pretend to always succeed. AFAIK, POSIX leaves it up to implementations
> to interpret concurrency levels, so making any level be equivalent to
> default level should be OK.

Yes to 1, No to 2, and the third option should be to treat THR_NEW_LWP
as creating a new system scope thread.

-- 
DE