8.2 + apache == a LOT of sigprocmask

[Doug Barton]

On 11/16/2011 23:49, Kostik Belousov wrote:
> On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote:
>> On 11/15/2011 02:09, Jeremy Chadwick wrote:
>>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote:
>>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote:
>>>>> On 11/14/2011 12:31, Doug Barton wrote:
>>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386
>>>>>> in a busy web hosting environment I came across the following post:
>>>>>>
>>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html
>>>>>>
>>>>>> That basically describes what we're seeing as well, including the
>>>>>> "doesn't happen on Linux" part.
>>>>>>
>>>>>> Does anyone have any ideas about this?
>>>>>>
>>>>>> With incredibly similar stuff running on 7.x we didn't see this problem,
>>>>>> so it seems to be something new in 8.
>>>>>
>>>>> Just took a closer look at our ktrace, and actually our pattern is
>>>>> slightly different than the one in that post. In ours the second option
>>>>> is null, but the third is set:
>>>>>
>>>>> 74195 httpd    0.000017 RET   sigprocmask 0
>>>>> 74195 httpd    0.000013 CALL  sigprocmask(SIG_BLOCK,0,0xbfbf89d4)
>>>>> 74195 httpd    0.000009 RET   sigprocmask 0
>>>>> 74195 httpd    0.000013 CALL  sigprocmask(SIG_BLOCK,0,0xbfbf89d4)
>>>>> 74195 httpd    0.000009 RET   sigprocmask 0
>>>>> 74195 httpd    0.000012 CALL  sigprocmask(SIG_BLOCK,0,0xbfbf89d4)
>>>>>
>>>>> But repeated hundreds of times in a row.
>>>>
>>>> The calls cannot come from rtld, they are generated by some setjmp()
>>>> invocation. If signal-safety is not needed, sigsetjmp() should be used
>>>> instead.
>>>>
>>>> Quick grep of the apache httpd source shows a single setjmp() in their
>>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0).
>>>
>>> I hate cross-posting, but: adding freebsd-apache@ to the list.  Some of
>>> the Apache folks (not just port committers) may have some insight to
>>> Kostik's findings.
>>
>> Thanks to everyone for the responses. We tried Kostik's suggestion and
>> unfortunately it didn't reduce the number of sigprocmask() calls to a
>> statistically significant degree.
>>
>> Does anyone have any other ideas on ways to debug this? We're sort of
>> running out of things to test. :-/
>>
>> Given how important (and prevalent) the Apache + FreeBSD combination is,
>> I'm kind of disturbed that we're seeing this performance problem, and if
>> it's something in 8.x that's also in 9.x, it would be better to fix it
>> prior to 9.0-RELEASE.
> 
> Since my guess appeared to be not useful,

Well I wouldn't say that they weren't useful, we eliminated the obvious
candidate. So, "not good news" certainly, but not unhelpful. :)

> the way forward is to identify
> the location of the call(s) that cause the issue. I suggest compliling
> at least apache itself, libc, rtld and libthr (if used) with debugging
> information. Then, attach to the running apache worker with the gdb and
> set breakpoint on sigprocmask. Several backtraces from the hit breakpoint
> should give enough data.

We tried that, and got this:

Loaded symbols for /libexec/ld-elf.so.1
0x28183a5d in accept () from /lib/libc.so.7
(gdb) b sigprocmask
Breakpoint 1 at 0x282d8f84
(gdb) c
Continuing.
no thread to satisfy query
0x28183a5d in accept () from /lib/libc.so.7
(gdb)

Of course I'm not the world's greatest gdb'er, so maybe there is a
better way to do it?

> High-tech solution is to link with libunwind and add code into sigprocmask()
> to gather the stacks. But I expect that gdb attach is enough.

Ok, we'll look into that, thanks.


Doug

-- 

		"We could put the whole Internet into a book."
		"Too practical."

	Breadth of IT experience, and depth of knowledge in the DNS.
	Yours for the right price.  :)  http://SupersetSolutions.com/