6-STABLE Fatal trap 12

[Alexandre Biancalana]

Hi list,

 We have a backup server Dell Power Edge 1850, One Xeon 3.6 Ghz, 2GB Ram,
attached to one storage PowerVault 220 and one LTO tape drive, running
STABLE from start of June.

This machine basically runs FTP Server, NFS Server, compress directories
using tar and gzip and run some perl scripts. Some from now this machine is
presenting random freezes, then I compiled a kernel with:

makeoptions     DEBUG=-g
options         KDB
options         DDB
options         INVARIANTS
options         INVARIANT_SUPPORT
options         WITNESS
options         DEBUG_LOCKS
options         DEBUG_VFS_LOCKS
options         DIAGNOSTIC


yesterday the machine freeze completely, no ping, no keyboard response, I
have to do a hard reset (hold power off button) to bring this back.

Today happened again and this time they break to debbuger and I can get a
dump that follow.

Excuse-me by sending images. This is because I don't have a serial cable to
plug on the machine, they are on a Datacenter and usually I have only remote
access.


http://www.seudns.net/~ale/crash/page_fault.jpg

http://www.seudns.net/~ale/crash/bt.jpg

http://www.seudns.net/~ale/crash/ps_1.jpg
http://www.seudns.net/~ale/crash/ps_2.jpg
http://www.seudns.net/~ale/crash/ps_3.jpg
http://www.seudns.net/~ale/crash/ps_4.jpg
http://www.seudns.net/~ale/crash/ps_5.jpg
http://www.seudns.net/~ale/crash/ps_6.jpg
http://www.seudns.net/~ale/crash/ps_7.jpg

http://www.seudns.net/~ale/crash/show_allcpu.jpg

http://www.seudns.net/~ale/crash/show_alllocks.jpg

http://www.seudns.net/~ale/crash/show_lockedvnodes.jpg
http://www.seudns.net/~ale/crash/show_lockedvnodes_2.jpg
http://www.seudns.net/~ale/crash/show_lockedvnodes_3.jpg


======== dmesg ========
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-STABLE #0: Wed Jun 13 19:07:04 BRT 2007
    root at Jones:/usr/src/sys/i386/compile/Jones
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
ACPI APIC Table: <DELL   PE BKC  >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.60GHz (3591.25-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf43  Stepping = 3

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SS
E2,SS,HTT,TM,PBE>
  Features2=0x659d<SSE3,RSVD2,MON,DS_CPL,EST,TM2,CNTX-ID,CX16,<b14>>
  AMD Features=0x20100000<NX,LM>
real memory  = 2147221504 (2047 MB)
avail memory = 2095714304 (1998 MB)
ioapic0: Changing APIC ID to 1
ioapic1: Changing APIC ID to 2
ioapic1: WARNING: intbase 32 != expected base 24
ioapic2: Changing APIC ID to 3
ioapic2: WARNING: intbase 64 != expected base 56
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 32-55 on motherboard
ioapic2 <Version 2.0> irqs 64-87 on motherboard
kbd1 at kbdmux0
acpi0: <DELL PE BKC> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
ahc0: <Adaptec 3960D Ultra160 SCSI adapter> port 0xec00-0xecff mem
0xfe9ff000-0xfe9fffff irq 39 at device 12.0 on pci2
ahc0: [GIANT-LOCKED]
aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
ahc1: <Adaptec 3960D Ultra160 SCSI adapter> port 0xe800-0xe8ff mem
0xfe9fe000-0xfe9fefff irq 36 at device 12.1 on pci2
ahc1: [GIANT-LOCKED]
aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs
amr0: <LSILogic MegaRAID 1.53> mem
0xf81f0000-0xf81fffff,0xfe9c0000-0xfe9dffff irq 46 at device 14.0 on pci2
amr0: delete logical drives supported by controller
amr0: <LSILogic PERC 4e/Si> Firmware 522A, BIOS H430, 256MB RAM
pcib3: <ACPI PCI-PCI bridge> at device 0.2 on pci1
pci3: <ACPI PCI bus> on pcib3
amr1: <LSILogic MegaRAID 1.53> mem 0xf80f0000-0xf80fffff irq 37 at device
11.0 on pci3
amr1: delete logical drives supported by controller
amr1: <LSILogic PERC 4/DC> Firmware 352B, BIOS 1.10, 128MB RAM
pcib4: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 0.0 on pci5
pci6: <ACPI PCI bus> on pcib6
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0xdcc0-0xdcff mem 0xfe5e0000-0xfe5fffff irq 64 at device 7.0
on pci6
em0: Ethernet address: 00:14:22:16:ba:37
pcib7: <ACPI PCI-PCI bridge> at device 0.2 on pci5
pci7: <ACPI PCI bus> on pcib7
em1: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0xccc0-0xccff mem 0xfe3e0000-0xfe3fffff irq 65 at device 8.0
on pci7
em1: Ethernet address: 00:14:22:16:ba:38
pcib8: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci8: <ACPI PCI bus> on pcib8
uhci0: <Intel 82801EB (ICH5) USB controller USB-A> port 0xace0-0xacff irq 16
at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <Intel 82801EB (ICH5) USB controller USB-A> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 82801EB (ICH5) USB controller USB-B> port 0xacc0-0xacdf irq 19
at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <Intel 82801EB (ICH5) USB controller USB-B> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 82801EB (ICH5) USB controller USB-C> port 0xaca0-0xacbf irq 18
at device 29.2 on pci0
uhci2: [GIANT-LOCKED]
usb2: <Intel 82801EB (ICH5) USB controller USB-C> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
ehci0: <Intel 82801EB/R (ICH5) USB 2.0 controller> mem 0xfeb00000-0xfeb003ff
irq 23 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
usb3: EHCI version 1.0
usb3: companion controllers, 2 ports each: usb0 usb1 usb2
usb3: <Intel 82801EB/R (ICH5) USB 2.0 controller> on ehci0
usb3: USB revision 2.0
uhub3: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub3: 6 ports with 6 removable, self powered
uhub4: vendor 0x413c product 0xa001, class 9/0, rev 2.00/0.00, addr 2
uhub4: multiple transaction translators
uhub4: 2 ports with 2 removable, self powered
ehci_device_intr_done: not done, ex=0xc64b5e00
ukbd0: Tripplite B015-000 R0.74 USB to PS2 adapter., rev 1.10/1.00, addr 3,
iclass 3/1
ehci_device_intr_done: not done, ex=0xc64d0d00
kbd2 at ukbd0
ums0: Tripplite B015-000 R0.74 USB to PS2 adapter., rev 1.10/1.00, addr 3,
iclass 3/1
ums0: 5 buttons and Z dir.
pcib9: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pci9: <display, VGA> at device 13.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH5 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on
acpi0
Timecounter "HPET" frequency 14318180 Hz quality 2000
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xec000-0xeffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 3591247239 Hz quality 800
Timecounters tick every 1.000 msec
Waiting 5 seconds for SCSI devices to settle
acd0: CDROM <TEAC CD-ROM CD-224E-N/3.AB> at ata0-master UDMA33
amr0: delete logical drives supported by controller
amrd0: <LSILogic MegaRAID logical drive> on amr0
amrd0: 69880MB (143114240 sectors) RAID 1 (optimal)
amr1: delete logical drives supported by controller
amrd1: <LSILogic MegaRAID logical drive> on amr1
amrd1: 1259100MB (2578636800 sectors) RAID 5 (optimal)
sa0 at ahc0 bus 0 target 6 lun 0
sa0: <CERTANCE ULTRIUM 2 1785> Removable Sequential Access SCSI-3 device
sa0: 160.000MB/s transfers (80.000MHz, offset 127, 16bit)
Trying to mount root from ufs:/dev/amrd0s1a
WARNING: / was not properly dismounted
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
/var: mount pending error: blocks 4 files 1
WARNING: /backup was not properly dismounted
Expensive timeout(9) function: 0xc0646fa8(0xc0775ee0) 0.011991048 s
em0: link state changed to UP

======= dmesg =========


I update system bios, perc 4/si and perc 4/dc firmware.

Any help is appreciated to discover if this is a hardware or software
problem.

Regards,

Alexandre