Fwd: Re: 2.3 upgrade on apache.org

[Philip M. Gollucci]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

FYI



- -------- Original Message --------
Subject: Re: 2.3 upgrade on apache.org
Date: Sun, 18 Jul 2010 10:58:45 -0700
From: Paul Querna <paul at querna.org>
Reply-To: dev at httpd.apache.org
To: dev at httpd.apache.org

A small update:

We thought the load average was a little high, in the 70+ range.  We
observed massive CPU usage from the httpd processes, but IO wise the
machine was fine -- tons of IO bandwidth to go around -- it seemed to
be something spinning inside httpd itself.

At first I thought it might of been the event mpm bug, so with the
awesome MPMs as a module, we switched to worker.

Didn't help.

I dug out ktrace, and started seeing what we were doing.

Every httpd processes I looked at was spinning on sendfile:

33893 httpd    CALL
sendfile(0x33,0x23,0x1d3000,0xbe6201,0x7ffffd1e9bd0,0x7ffffd1e9bf8,<invalid=0>)
33893 httpd    RET sendfile -1 errno 35 Resource temporarily unavailable

ktrace/dtrace dumps available to any committer who wants to take a
look.  A similar problem is described in this thread from 2002, about
running 2.0.x on Daedalus:
  <http://www.mail-archive.com/dev@httpd.apache.org/msg04928.html>

I've started tuning up the FreeBSD sysctls for network buffers etc,
but it didn't seem to have a significant effect.

We have now disabled Sendfile on apache.org, and the load average
dropped from ~80 to 0.35.

On Sun, Jul 18, 2010 at 3:08 AM, Paul Querna <paul at querna.org> wrote:
> www.apache.org is now running trunk @ r965127, using the Event MPM on
> FreeBSD 8.1, on a new x86 box.
>
> Previously it was running on Solaris 10, with 2.2.x on sparc t2000s.
>
> I have also enabled OCSP stapling on the SSL side:
>  <https://www.apache.org/>
>
> Anyways, I've made a list of the 2.3-ish issues seen during the upgrade.
>
> httpd itself:
>  - Using the Define directive to enable SSL was a bad idea.  The SSL
> init code was only being called once in post_config instead of twice
> This is only 'new' because the Define directive is new, so I tried to
> use it.  Example of how it was used:
>  <IfDefine MACHINE_NAME>
>     Define SSL
>     Listen 1.2.3.4:
>  </IfDefine>
>
>  <IfDefine SSL>
>    LoadModule ssl_module ..
>  </IfDefine>
>
> We do -D MACHINE_NAME at the command line, and I was hoping to just
> enable SSL from within the config file, but that didn't work. Adding
> -D SSL to the command line fixed the issue.
>
> - Lots of warnings in our configurations about "Ignoring deprecated
> use of DefaultType".
>
> - Infra already had lots of <IfVersion blocks for most 2.3 authn/z
> changes, and those seem to stil be working fine.
>
> 3rd party:
>
> - APR's detection of BDB was funky again, I had --with-dbm=db48, which
> it was happy about, but until I added --with-berkeley-db=/usr/local,
> it wouldn't actually compile up the shared objects for the DB DBM, but
> configure reported success.
>
> - mod_wsgi:  Needed to comment out 2 lines debug logging, that were
> accessing server_rec->loglevel (which has been removed from 2.3.x).
>
> - mod_mbox/mod_allowmethods/mod_asf_cgi: Needed build system
> adjustments to not look for apr-util.
>
> Surprisingly painless upgrade after I figured out the mod_ssl config
> issue; Most of the work was around moving Solaris -> FreeBSD for our
> various rsyncs and crontabs,  of course let infra know if you see
> anything busted!
>
> Thanks,
>
> Paul
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (FreeBSD)

iD8DBQFMRNSRdbiP+9ubjBwRAippAJ49ODp6c+w8RsGTKYVf0yQblTbpYwCfSt6c
od7KirxvHpV2dPTcWxi6wyI=
=wrVg
-----END PGP SIGNATURE-----