8.2 + apache == a LOT of sigprocmask

[Doug Barton]

On 11/17/2011 00:12, Kostik Belousov wrote:
> On Wed, Nov 16, 2011 at 11:59:06PM -0800, Doug Barton wrote:
>> On 11/16/2011 23:49, Kostik Belousov wrote:
>>> On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote:
>>>> On 11/15/2011 02:09, Jeremy Chadwick wrote:
>>>>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote:
>>>>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote:
>>>>>>> On 11/14/2011 12:31, Doug Barton wrote:
>>>>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386
>>>>>>>> in a busy web hosting environment I came across the following post:
>>>>>>>>
>>>>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html
>>>>>>>>
>>>>>>>> That basically describes what we're seeing as well, including the
>>>>>>>> "doesn't happen on Linux" part.
>>>>>>>>
>>>>>>>> Does anyone have any ideas about this?
>>>>>>>>
>>>>>>>> With incredibly similar stuff running on 7.x we didn't see this problem,
>>>>>>>> so it seems to be something new in 8.
>>>>>>>
>>>>>>> Just took a closer look at our ktrace, and actually our pattern is
>>>>>>> slightly different than the one in that post. In ours the second option
>>>>>>> is null, but the third is set:
>>>>>>>
>>>>>>> 74195 httpd    0.000017 RET   sigprocmask 0
>>>>>>> 74195 httpd    0.000013 CALL  sigprocmask(SIG_BLOCK,0,0xbfbf89d4)
>>>>>>> 74195 httpd    0.000009 RET   sigprocmask 0
>>>>>>> 74195 httpd    0.000013 CALL  sigprocmask(SIG_BLOCK,0,0xbfbf89d4)
>>>>>>> 74195 httpd    0.000009 RET   sigprocmask 0
>>>>>>> 74195 httpd    0.000012 CALL  sigprocmask(SIG_BLOCK,0,0xbfbf89d4)
>>>>>>>
>>>>>>> But repeated hundreds of times in a row.
>>>>>>
>>>>>> The calls cannot come from rtld, they are generated by some setjmp()
>>>>>> invocation. If signal-safety is not needed, sigsetjmp() should be used
>>>>>> instead.
>>>>>>
>>>>>> Quick grep of the apache httpd source shows a single setjmp() in their
>>>>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0).
>>>>>
>>>>> I hate cross-posting, but: adding freebsd-apache@ to the list.  Some of
>>>>> the Apache folks (not just port committers) may have some insight to
>>>>> Kostik's findings.
>>>>
>>>> Thanks to everyone for the responses. We tried Kostik's suggestion and
>>>> unfortunately it didn't reduce the number of sigprocmask() calls to a
>>>> statistically significant degree.
>>>>
>>>> Does anyone have any other ideas on ways to debug this? We're sort of
>>>> running out of things to test. :-/
>>>>
>>>> Given how important (and prevalent) the Apache + FreeBSD combination is,
>>>> I'm kind of disturbed that we're seeing this performance problem, and if
>>>> it's something in 8.x that's also in 9.x, it would be better to fix it
>>>> prior to 9.0-RELEASE.
>>>
>>> Since my guess appeared to be not useful,
>>
>> Well I wouldn't say that they weren't useful, we eliminated the obvious
>> candidate. So, "not good news" certainly, but not unhelpful. :)
>>
>>> the way forward is to identify
>>> the location of the call(s) that cause the issue. I suggest compliling
>>> at least apache itself, libc, rtld and libthr (if used) with debugging
>>> information. Then, attach to the running apache worker with the gdb and
> Note this part.

Right, we attached to a worker, that's why it's in accept(). :)

> It seems your libc has no debugging information.
> accept() is the pure syscall wrapper, it cannot call sigprocmask.
> If gdb catched the PLT trampoline instead of real accept(),  we would
> see the rtld frames. So install libc, libthr and rtld with debug.

It's not catching there though:

Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
0x28183b2d in accept () at accept.S:3
3	RSYSCALL(accept)
(gdb) c
Continuing.
no thread to satisfy query
0x28183b2d in accept () at accept.S:3
3	RSYSCALL(accept)
(gdb) info threads
Cannot get thread info: invalid key
(gdb)



Doug

-- 

		"We could put the whole Internet into a book."
		"Too practical."

	Breadth of IT experience, and depth of knowledge in the DNS.
	Yours for the right price.  :)  http://SupersetSolutions.com/