www/128689: Fatal trap 12: page fault while in kernel mode from load on Apache/PHP Jail

[Tristan Waddington]

>Number:         128689
>Category:       www
>Synopsis:       Fatal trap 12: page fault while in kernel mode from load on Apache/PHP Jail
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-www
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Nov 08 07:50:01 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator:     Tristan Waddington
>Release:        FreeBSD 7.0-RELEASE
>Organization:
University of Oregon
>Environment:
FreeBSD drone0.uoregon.edu 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Sun Feb 24 19:59:52 UTC 2008     root at logan.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386
>Description:
Running several Apache 2.2.9/PHP 5.2.6 Jails on a FreeBSD 7.0-RELEASE server.  Kind of a complex setup.  Been running rock solid for several months, but just recently ran into some unexpected Kernel Panics.  To troubleshoot we put some load on some of our Jails (ezjail) with Apache AB and every single time were able to get the server to completely lock up with this error (core dump backtrace follows):

[root at drone0 /boot/kernel]# kgdb kernel.symbols /var/crash/vmcore.0
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 03
fault virtual address    = 0x0
fault code        = supervisor read, page not present
instruction pointer    = 0x20:0xc0744ad7
stack pointer            = 0x28:0xe9af8b30
frame pointer            = 0x28:0xe9af8b88
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, def32 1, gran 1
processor eflags    = resume, IOPL = 0
current process        = 49963 (httpd)
trap number        = 12
panic: page fault
cpuid = 3
Uptime: 60d6h36m34s
Physical memory: 3567 MB
Dumping 328 MB: 313 297 281 265 249 233 217 201 185 169 153 137 121 105 89 73 57 41 25 9

#0  doadump () at pcpu.h:195
195    pcpu.h: No such file or directory.
    in pcpu.h
(kgdb) backtrace
#0  doadump () at pcpu.h:195
#1  0xc0754457 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0754719 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xc0a4905c in trap_fatal (frame=0xe9af8af0, eva=0) at /usr/src/sys/i386/i386/trap.c:899
#4  0xc0a499df in trap (frame=0xe9af8af0) at /usr/src/sys/i386/i386/trap.c:280
#5  0xc0a2fc0b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#6  0xc0744ad7 in lf_advlock (ap=0xe9af8c20, head=0xc9a3e3d0, size=0) at /usr/src/sys/kern/kern_lockf.c:294
#7  0xc095d7ad in ufs_advlock (ap=0xe9af8c20) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2181
#8  0xc0a5e1e7 in VOP_ADVLOCK_APV (vop=0xc0b93c60, a=0xe9af8c20) at vnode_if.c:1977
#9  0xc0729547 in kern_fcntl (td=0xc7fce840, fd=16, cmd=9, arg=-374371232) at vnode_if.h:1036
#10 0xc0729e07 in fcntl (td=0xc7fce840, uap=0xe9af8cfc) at /usr/src/sys/kern/kern_descrip.c:336
#11 0xc0a49635 in syscall (frame=0xe9af8d38) at /usr/src/sys/i386/i386/trap.c:1035
#12 0xc0a2fc70 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:196
#13 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)

Now, we did find out that the custom PHP framework we're running is a little inefficient.  But we tried the same AB load test on a Linux box we have and it didn't crash.  It just slowed down a lot for a minute.

Now if I'm not mistaken, FreeBSD should be pretty stable, despite the poor scripting.  So I'm thinking something with our configuration must be making it crash.  Either that or there is a bug present somewhere in the packages we're running.

We also tried the same test on a phpBB3 Jail that we're running (not our framework) and it didn't take down the system. Still, I'm not convinced it could completely be caused by a bad PHP script.  That would be bad mojo.

PHP and Apache were both installed from the ports tree.  These are the packages our typical Jail has:

apache-2.2.9        Version 2.2 of Apache web server with prefork MPM.
aspell-0.60.6_2     Spelling checker with better suggestion logic than ispell
autoconf-2.61_2     Automatically configure source code on many Un*x platforms
autoconf-wrapper-20071109 Wrapper script for GNU autoconf
bash-3.2.25         The GNU Project's Bourne Again SHell
bigreqsproto-1.0.2  BigReqs extension headers
curl-7.18.0         Non-interactive tool to get files from FTP, GOPHER, HTTP(S)
expat-2.0.1         XML 1.0 parser written in C
freetype2-2.3.7     A free and portable TrueType font rendering engine
gettext-0.16.1_3    GNU gettext package
gmake-3.81_3        GNU version of 'make' utility
help2man-1.36.4_2   Automatically generating simple manual pages from program o
inputproto-1.4.2.1  Input extension headers
jpeg-6b_4           IJG's jpeg compression utilities
kbproto-1.0.3       KB extension headers
libICE-1.0.4_1,1    Inter Client Exchange library for X11
libSM-1.0.3_1,1     Session Management library for X11
libX11-1.1.3_1,1    X11 library
libXau-1.0.3_2      Authentication Protocol library for X11
libXaw-1.0.4_1,1    X Athena Widgets library
libXdmcp-1.0.2_1    X Display Manager Control Protocol library
libXext-1.0.3,1     X11 Extension library
libXmu-1.0.3,1      X Miscellaneous Utilities libraries
libXp-1.0.0,1       X print library
libXpm-3.5.7        X Pixmap library
libXt-1.0.5_1       X Toolkit library
libiconv-1.11_1     A character set conversion library
libltdl-1.5.26      System independent dlopen wrapper
libmcrypt-2.5.8     Multi-cipher cryptographic library (used in PHP)
libtool-1.5.26      Generic shared library support script
libxml2-2.6.32      XML parser library for GNOME
lynx-2.8.6.5_4,1    A non-graphical, text-based World-Wide Web client
m4-1.4.11,1         GNU m4
mhash-0.9.9         An easy-to-use library for strong hashes such as MD5 and SH
mysql-client-5.0.51a Multithreaded SQL database (client)
nano-2.0.6          Nano's ANOther editor, an enhanced free Pico clone
p5-gettext-1.05_2   Message handling functions
pcre-7.7            Perl Compatible Regular Expressions library
pdflib-7.0.2        A C library for dynamically generating PDF
pecl-APC-3.0.19     Alternative PHP Cache
pecl-fileinfo-1.0.4 A PECL extension to retrieve info about files
pecl-filter-0.11.0  PHP extension for safely dealing with input parameters
pecl-hash-1.5       HASH Message Digest Framework for PHP
pecl-json-1.2.1     PHP extension for JSON (JavaScript Object Notation) seriali
pecl-pdflib-2.1.5   A PECL extension to create PDF on the fly
pecl-zip-1.9.0      A PECL extension to create and read zip files
perl-5.8.8_1        Practical Extraction and Report Language
php5-5.2.6          PHP Scripting Language
php5-calendar-5.2.6 The calendar shared extension for php
php5-ctype-5.2.6    The ctype shared extension for php
php5-curl-5.2.6     The curl shared extension for php
php5-dom-5.2.6      The dom shared extension for php
php5-exif-5.2.6     The exif shared extension for php
php5-extensions-1.1 A "meta-port" to install PHP extensions
php5-ftp-5.2.6      The ftp shared extension for php
php5-gd-5.2.6       The gd shared extension for php
php5-gettext-5.2.6_1 The gettext shared extension for php
php5-iconv-5.2.6    The iconv shared extension for php
php5-mbstring-5.2.6 The mbstring shared extension for php
php5-mcrypt-5.2.6   The mcrypt shared extension for php
php5-mhash-5.2.6    The mhash shared extension for php
php5-mysql-5.2.6    The mysql shared extension for php
php5-openssl-5.2.6  The openssl shared extension for php
php5-pcre-5.2.6     The pcre shared extension for php
php5-pdo-5.2.6      The pdo shared extension for php
php5-pdo_sqlite-5.2.6 The pdo_sqlite shared extension for php
php5-posix-5.2.6    The posix shared extension for php
php5-pspell-5.2.6_1 The pspell shared extension for php
php5-session-5.2.6  The session shared extension for php
php5-simplexml-5.2.6 The simplexml shared extension for php
php5-sockets-5.2.6  The sockets shared extension for php
php5-spl-5.2.6      The spl shared extension for php
php5-sqlite-5.2.6   The sqlite shared extension for php
php5-tokenizer-5.2.6 The tokenizer shared extension for php
php5-xml-5.2.6      The xml shared extension for php
php5-xmlreader-5.2.6 The xmlreader shared extension for php
php5-xmlwriter-5.2.6 The xmlwriter shared extension for php
php5-zlib-5.2.6     The zlib shared extension for php
pkg-config-0.23_1   A utility to retrieve information about installed libraries
png-1.2.28          Library for manipulating PNG images
printproto-1.0.3    Print extension headers
rsync-3.0.3         A network file distribution/synchronization utility
sudo-1.6.9.6        Allow others to run commands as root
t1lib-5.1.2,1       A Type 1 Rasterizer Library for UNIX/X11
xcmiscproto-1.1.2   XCMisc extension headers
xextproto-7.0.2     XExt extension headers
xf86bigfontproto-1.1.2 XFree86-Bigfont extension headers
xproto-7.0.10_1     X11 protocol headers
xtrans-1.0.4        Abstract network code for X

Any advice you can give me would be helpful.  Thanks!
>How-To-Repeat:
Run AB with 200 concurrent connections on our Apache/PHP jail running our PHP framework.
>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: