5.4/amd64 not stable

Tue Jan 17 23:13:23 PST 2006

Hello list,

I am observing panics of FreeBSD 5.4/amd64
on a web server. We tested it heavily with
Apache bench. Load was over 250, without problems.
However, when we put the server in production,
it crashes time after time. It takes some time
between 1 and 8 hours. The panics look like this:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x48
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xffffffff80271a83
stack pointer           = 0x10:0xffffffffc1043990
frame pointer           = 0x10:0x0
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 10567 (httpd)
trap number             = 12
panic: page fault
cpuid = 0
boot() called on cpu#0

The instruction pointer is always the same, and process is always httpd.
Does it mean anything?

We are not able to produce crash dump. Dumping memory into swap
starts after crash, but it stops after 4GB and no further progress
is apparent. If we break it, savecore does not recognize the data
in swap space as crash dump during boot.

BTW, it is HP ProLiant DL385, dmesg.boot and kernel configuration
are attached. It has got: 2 Opterons, 16GB RAM, it runs Apache with
mod_ssl and php, a mysql server, a nfs server and a jail. I am attaching
pkg_info as well.

Both memtest and HP diagnostic tools show no hardware problems.

I would appreciate any ideas or comments. Thanks in advance,
--
Josef
-------------- next part --------------
Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD 5.4-RELEASE-p8 #0: Wed Nov 16 10:50:18 CET 2005
    root at amd64.tns.cz:/usr/obj/usr/src/sys/PLDL385
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Opteron(tm) Processor 252 (2605.92-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x20f51  Stepping = 1
  Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
  Features2=0x1<SSE3>
  AMD Features=0xe2500800<SYSCALL,NX,MMX+,<b25>,LM,3DNow+,3DNow>
real memory  = 17179865088 (16383 MB)
avail memory = 16480096256 (15716 MB)
ACPI APIC Table: <HP     00000083>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
ioapic1 <Version 1.1> irqs 24-27 on motherboard
ioapic2 <Version 1.1> irqs 28-31 on motherboard
ioapic3 <Version 1.1> irqs 32-35 on motherboard
ioapic4 <Version 1.1> irqs 36-39 on motherboard
acpi0: <HP A05> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci1: <base peripheral> at device 2.0 (no driver attached)
pci1: <base peripheral> at device 2.2 (no driver attached)
pci1: <display, VGA> at device 3.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 4.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <AMD 8111 UDMA133 controller> port 0x2000-0x200f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 at device 4.1 on pci0
ata0: channel #0 on atapci0
ata1: channel #1 on atapci0
pci0: <bridge> at device 4.3 (no driver attached)
pcib2: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci2: <ACPI PCI bus> on pcib2
ciss0: <HP Smart Array 6i> port 0x5000-0x50ff mem 0xf7c80000-0xf7cbffff,0xf7cf0000-0xf7cf1fff irq 24 at device 4.0 on pci2
pci0: <base peripheral, interrupt controller> at device 7.1 (no driver attached)
pcib3: <ACPI PCI-PCI bridge> at device 8.0 on pci0
pci3: <ACPI PCI bus> on pcib3
bge0: <Broadcom BCM5704C Dual Gigabit Ethernet, ASIC rev. 0x2100> mem 0xf7df0000-0xf7dfffff irq 28 at device 6.0 on pci3
miibus0: <MII bus> on bge0
brgphy0: <BCM5704 10/100/1000baseTX PHY> on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto
bge0: Ethernet address: 00:14:c2:3d:d3:0c
bge1: <Broadcom BCM5704C Dual Gigabit Ethernet, ASIC rev. 0x2100> mem 0xf7de0000-0xf7deffff irq 29 at device 6.1 on pci3
miibus1: <MII bus> on bge1
brgphy1: <BCM5704 10/100/1000baseTX PHY> on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto
bge1: Ethernet address: 00:14:c2:3d:d3:0b
pci0: <base peripheral, interrupt controller> at device 8.1 (no driver attached)
pcib4: <ACPI Host-PCI bridge> on acpi0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 9.0 on pci4
pci5: <ACPI PCI bus> on pcib5
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x6000-0x60ff mem 0xf7fc0000-0xf7fdffff,0xf7fe0000-0xf7ffffff irq 34 at device 7.0 on pci5
mpt1: <LSILogic 1030 Ultra4 Adapter> port 0x6400-0x64ff mem 0xf7f80000-0xf7f9ffff,0xf7fa0000-0xf7fbffff irq 35 at device 7.1 on pci5
em0: <Intel(R) PRO/1000 Network Connection, Version - 1.7.35> port 0x6800-0x683f mem 0xf7f00000-0xf7f3ffff,0xf7f60000-0xf7f7ffff irq 32 at device 8.0 on pci5
em0: Ethernet address: 00:11:0a:53:f2:c2
em0:  Speed:N/A  Duplex:N/A
em1: <Intel(R) PRO/1000 Network Connection, Version - 1.7.35> port 0x6840-0x687f mem 0xf7ee0000-0xf7efffff irq 33 at device 8.1 on pci5
em1: Ethernet address: 00:11:0a:53:f2:c3
em1:  Speed:N/A  Duplex:N/A
pci4: <base peripheral, interrupt controller> at device 9.1 (no driver attached)
pcib6: <ACPI PCI-PCI bridge> at device 10.0 on pci4
pci6: <ACPI PCI bus> on pcib6
pci4: <base peripheral, interrupt controller> at device 10.1 (no driver attached)
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
sio0: <Standard PC COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
fdc0: <floppy drive controller (FDE)> port 0x3f2-0x3f5 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
fdc0: <floppy drive controller (FDE)> port 0x3f2-0x3f5 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
orm0: <ISA Option ROMs> at iomem 0xee000-0xeffff,0xcc800-0xcdfff,0xcc000-0xcc7ff,0xc8000-0xcbfff,0xc0000-0xc7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
IP Filter: v3.4.35 initialized.  Default = pass all, Logging = enabled
acd0: DVDROM <DV-28E-N/C.6B> at ata0-master PIO4
Waiting 15 seconds for SCSI devices to settle
da0 at ciss0 bus 0 target 0 lun 0
da0: <COMPAQ RAID 5  VOLUME OK> Fixed Direct Access SCSI-0 device 
da0: 135.168MB/s transfers
da0: 138919MB (284506560 512 byte sectors: 255H 32S/T 34866C)
SMP: AP CPU #1 Launched!
Mounting root from ufs:/dev/da0s1a
-------------- next part --------------
machine		amd64
cpu		HAMMER
ident		PLDL385

# To statically compile in device wiring instead of /boot/device.hints
#hints		"GENERIC.hints"		# Default places to look for devices.

options 	SCHED_4BSD		# 4BSD scheduler
options 	INET			# InterNETworking
options 	FFS			# Berkeley Fast Filesystem
options 	SOFTUPDATES		# Enable FFS soft updates support
options 	UFS_ACL			# Support for access control lists
options 	UFS_DIRHASH		# Improve performance on big directories
options 	MD_ROOT			# MD is a potential root device
options 	NFSCLIENT		# Network Filesystem Client
options 	NFSSERVER		# Network Filesystem Server
options 	NFS_ROOT		# NFS usable as /, requires NFSCLIENT
options 	MSDOSFS			# MSDOS Filesystem
options 	CD9660			# ISO 9660 Filesystem
options 	PROCFS			# Process filesystem (requires PSEUDOFS)
options 	PSEUDOFS		# Pseudo-filesystem framework
options 	GEOM_GPT		# GUID Partition Tables.
options 	COMPAT_43		# Needed by COMPAT_LINUX32
options 	COMPAT_IA32		# Compatible with i386 binaries
options 	COMPAT_FREEBSD4		# Compatible with FreeBSD4
options 	COMPAT_LINUX32		# Compatible with i386 linux binaries 
options 	SCSI_DELAY=15000	# Delay (in ms) before probing SCSI
options 	KTRACE			# ktrace(1) support
options 	SYSVSHM			# SYSV-style shared memory
options 	SYSVMSG			# SYSV-style message queues
options 	SYSVSEM			# SYSV-style semaphores
options 	_KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options 	KBD_INSTALL_CDEV	# install a CDEV entry in /dev
options 	ADAPTIVE_GIANT		# Giant mutex is adaptive.

# Workarounds for some known-to-be-broken chipsets (nVidia nForce3-Pro150)
device		atpic		# 8259A compatability

options 	NO_MIXED_MODE	# Don't penalize working chipsets

# Bus support.  Do not remove isa, even if you have no isa slots
device		acpi
device		isa
device		pci

# Floppy drives
device		fdc

# ATA and ATAPI devices
device		ata
device		atadisk
device		atapicd		# ATAPI CDROM drives
options 	ATA_STATIC_ID	# Static device numbering

# SCSI Controllers
device		mpt		# LSI-Logic MPT-Fusion

# SCSI peripherals
device		scbus		# SCSI bus (required for SCSI)
device		ch		# SCSI media changers
device		da		# Direct Access (disks)
device		sa		# Sequential Access (tape etc)
device		cd		# CD
device		pass		# Passthrough device (direct SCSI access)
device		ses		# SCSI Environmental Services (and SAF-TE)

# RAID controllers interfaced to the SCSI subsystem
device		ciss		# Compaq Smart RAID 5*

# atkbdc0 controls both the keyboard and the PS/2 mouse
device		atkbdc		# AT keyboard controller
device		atkbd		# AT keyboard
device		psm		# PS/2 mouse

device		vga		# VGA video card driver

device		splash		# Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device		sc

# Serial (COM) ports
device		sio		# 8250, 16[45]50 based serial ports

# PCI Ethernet NICs.
device		em		# Intel PRO/1000 adapter Gigabit Ethernet Card

# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device		miibus		# MII bus support
device		bge		# Broadcom BCM570xx Gigabit Ethernet

# Pseudo devices.
device		loop		# Network loopback
device		mem		# Memory and kernel memory devices
device		io		# I/O device
device		random		# Entropy device
device		ether		# Ethernet support
device		tun		# Packet tunnel.
device		tap		# Virtual ethernet
device		pty		# Pseudo-ttys (telnet etc)
device		md		# Memory "disks"
device		gif		# IPv6 and IPv4 tunneling
device		faith		# IPv6-to-IPv4 relaying (translation)

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device		bpf		# Berkeley packet filter

options		IPFILTER
options		IPFILTER_LOG

options		PQ_CACHESIZE=1024
options		SMP
-------------- next part --------------
abcache-0.14        A tool to cache applications written in PHP
apache+mod_ssl-1.3.34+2.8.25_1 The Apache 1.3 webserver with SSL/TLS functionality
autoconf-2.59_2     Automatically configure source code on many Un*x platforms 
bash-3.0.16_1       The GNU Project's Bourne Again SHell
cvsup-without-gui-16.1h_2 General network file distribution system optimized for CVS 
expat-1.95.8_3      XML 1.0 parser written in C
freetype2-2.1.10_1  A free and portable TrueType font rendering engine
gd-2.0.33_3,1       A graphics library for fast creation of images
gettext-0.14.5      GNU gettext package
gmake-3.80_2        GNU version of 'make' utility
help2man-1.36.3     Automatically generating simple manual pages from program o
jpeg-6b_3           IJG's jpeg compression utilities
libiconv-1.9.2_1    A character set conversion library
libtool-1.5.20      Generic shared library support script (1.5)
m4-1.4.4            GNU m4
mm-1.4.0            Shared memory allocation library for pre-forked process mod
mod_security-1.8.7  An intrusion detection and prevention engine
mysql-client-4.0.26 Multithreaded SQL database (client)
mysql-scripts-4.0.26 Multithreaded SQL database (scripts)
mysql-server-4.0.26 Multithreaded SQL database (server)
p5-DBD-mysql40-3.0002 MySQL 4.0 driver for the Perl5 Database Interface (DBI)
p5-DBI-1.48         The perl5 Database Interface.  Required for DBD::* modules
p5-gettext-1.05_1   Message handling functions
pcre-6.4            Perl Compatible Regular Expressions library
perl-5.8.6_2        Practical Extraction and Report Language
php4-4.4.1_2        PHP Scripting Language (Apache Module and CLI)
php4-extensions-1.0 A "meta-port" to install PHP extensions
php4-gd-4.4.1_2     The gd shared extension for php
php4-iconv-4.4.1_2  The iconv shared extension for php
php4-mysql-4.4.1_2  The mysql shared extension for php
php4-pcre-4.4.1_2   The pcre shared extension for php
php4-session-4.4.1_2 The session shared extension for php
php4-xml-4.4.1_2    The xml shared extension for php
pkgconfig-0.20      A utility to retrieve information about installed libraries
png-1.2.8_2         Library for manipulating PNG images
portaudit-0.5.10    Checks installed ports against a list of security vulnerabi
postfix-2.2.5_1,1   A secure alternative to widely-used Sendmail
rsync-2.6.6         A network file distribution/synchronization utility
scponly-4.3         A tiny shell that only permits scp and sftp
t1lib-5.1.0,1       A Type 1 Rasterizer Library for UNIX/X11
vsftpd-2.0.3_1      A FTP daemon that aims to be "very secure"
webalizer-2.1.10_5  A web server log file analysis program
wget-1.10.2         Retrieve files from the Net via HTTP and FTP