Tracking down a problem with php on FreeBSD

[David Xu]

Ivan Voras wrote:
> On 5 February 2011 19:43, Ruslan Mahmatkhanov <cvs-src at yandex.ru> wrote:
>> Hi, Ivan!
>>
>> Thank you much for response and sorry for late answer. We was able to
>> collect some data about the issue to make discussion more objective. See
>> below.
> 
>>>> Simple php-fpm restart solves the problem, but i need to track it down
>>>> to the cause of this situation and ask for your assistance and
>>>> instructions on how to debug it. Some facts about this:
>>> On one hand, FPM is said to be very experimental...
>>>
>>> Personally, I've been using apache22-worker or apache22-event +
>>> mod_fcgid for years without trouble.
>> We prefer to avoid using apache at all, because in this it's just adds yet
>> another unneeded link and complexity.
> 
> I guess it's about tradeoffs beween complexity and stability :)
> 
>>>> - `top -mio` shows very high (80000-90000 for VCSW) VCSW/IVCSW values
>>>> for php-fpm processes and LA is more than 120
> 
> I think this is significant, especially with this:
> 
>> When attaching to any hanging php-fpm proccess with truss, than i see a lot
>> of this calls:
>> sched_yield(0x80516c000,0x1,0x4d4828b6,0x8012ef45c,0xffffffff808bfd80,0x7fffffffa078)
>> = 0 (0x0)
>> sched_yield(0x80516c000,0x1,0x4d4828b6,0x8012ef45c,0xffffffff808bfd80,0x7fffffffa078)
>> = 0 (0x0)
>> sched_yield(0x80516c000,0x1,0x4d4828b6,0x8012ef45c,0xffffffff808bfd80,0x7fffffffa078)
>> = 0 (0x0)
>> sched_yield(0x80516c000,0x1,0x4d4828b6,0x8012ef45c,0xffffffff808bfd80,0x7fffffffa078)
>> = 0 (0x0)
> 
> "Normal" processes of the type PHP is have no need to call
> sched_yield() arbitrarily, unless they are implementing something they
> shouldn't - like a synchronization primitive (semaphore/lock). If "a
> lot" means "of the same order of magnitude as your VCSW rate", this is
> the reason for it.
> 
> I've analyzed my php-cgi binary and modules and they don't use sched_yield.
> 
> And yes, grepping for it in the source finds it only in FPM:
> 
> sapi/fpm/fpm/fpm_atomic.h:140:          sched_yield();
> 
> It seems they are trying to implement a spinlock by hand, instead of
> using what the OS provides. (on the other hand, the implementation
> might be correct but they may be using it wrong).
> 
> In any case, this points to bugs in FPM. if so, unfortunately I can't
> help you further.
> 
> If you really want to continue using FPM, I guess you should probably
> replace this hand-made lock implementation by sem(4) or see if
> "robust" pthreads mutexes can be committed and MFCed (maybe with David
> Xu).
>
Yes, there is a p4 branch implemented pthread robust mutex, it requires
ABI change.
Document for the POSIX robust mutex is here:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_mutexattr_getrobust.html