Tracking down a problem with php on FreeBSD

[Ivan Voras]

On 5 February 2011 19:43, Ruslan Mahmatkhanov <cvs-src at yandex.ru> wrote:
> Hi, Ivan!
>
> Thank you much for response and sorry for late answer. We was able to
> collect some data about the issue to make discussion more objective. See
> below.

>>> Simple php-fpm restart solves the problem, but i need to track it down
>>> to the cause of this situation and ask for your assistance and
>>> instructions on how to debug it. Some facts about this:
>>
>> On one hand, FPM is said to be very experimental...
>>
>> Personally, I've been using apache22-worker or apache22-event +
>> mod_fcgid for years without trouble.
>
> We prefer to avoid using apache at all, because in this it's just adds yet
> another unneeded link and complexity.

I guess it's about tradeoffs beween complexity and stability :)

>>> - `top -mio` shows very high (80000-90000 for VCSW) VCSW/IVCSW values
>>> for php-fpm processes and LA is more than 120

I think this is significant, especially with this:

> When attaching to any hanging php-fpm proccess with truss, than i see a lot
> of this calls:
> sched_yield(0x80516c000,0x1,0x4d4828b6,0x8012ef45c,0xffffffff808bfd80,0x7fffffffa078)
> = 0 (0x0)
> sched_yield(0x80516c000,0x1,0x4d4828b6,0x8012ef45c,0xffffffff808bfd80,0x7fffffffa078)
> = 0 (0x0)
> sched_yield(0x80516c000,0x1,0x4d4828b6,0x8012ef45c,0xffffffff808bfd80,0x7fffffffa078)
> = 0 (0x0)
> sched_yield(0x80516c000,0x1,0x4d4828b6,0x8012ef45c,0xffffffff808bfd80,0x7fffffffa078)
> = 0 (0x0)

"Normal" processes of the type PHP is have no need to call
sched_yield() arbitrarily, unless they are implementing something they
shouldn't - like a synchronization primitive (semaphore/lock). If "a
lot" means "of the same order of magnitude as your VCSW rate", this is
the reason for it.

I've analyzed my php-cgi binary and modules and they don't use sched_yield.

And yes, grepping for it in the source finds it only in FPM:

sapi/fpm/fpm/fpm_atomic.h:140:          sched_yield();

It seems they are trying to implement a spinlock by hand, instead of
using what the OS provides. (on the other hand, the implementation
might be correct but they may be using it wrong).

In any case, this points to bugs in FPM. if so, unfortunately I can't
help you further.

If you really want to continue using FPM, I guess you should probably
replace this hand-made lock implementation by sem(4) or see if
"robust" pthreads mutexes can be committed and MFCed (maybe with David
Xu).

Here is the FPM file:

http://svn.php.net/viewvc/php/php-src/branches/PHP_5_3/sapi/fpm/fpm/fpm_atomic.h?revision=305417&view=markup