fatal trap 12 in pagedaemon on dual-core opteron machine

Rob Watt rob at hudson-trading.com
Thu Jun 30 20:00:50 GMT 2005


I've been having stability problems in a new dual-core opteron machine.
When it first crashed we were running a number of multicast
applications that were listening to, recording, and rebroadcasting data. I
have been able to re-simulate the data play upto the crash and can
recreate the panic 1 out of every 5 or 6 replays.

the hardware is:

- 2 275 (2.2Ghz) dual-core processors
- tyan 2881 k8sr motherboard (bios 2.05v)
- 2 Gb DDR 400Mhz PC3200 Registered ECC memory
- SEAGATE ST3146807LC scsi drive
- intel dual-port gigabit ethernet card

I have experienced the problem in both 5.4-RELEASE and STABLE (stable
as-of today). I am running a custom SMP kernel.

The most recent panic message was:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x88
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xffffffff802af0da
stack pointer           = 0x10:0xffffffffb1c16a60
frame pointer           = 0x10:0xffffff005b8cd000
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 63 (pagedaemon)
trap number             = 12
panic: page fault
cpuid = 0
boot() called on cpu#0
Uptime: 1h3m10s
Dumping 2047 MB
 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320
336 352 368 384 400 416 432 448 464 480 496 512 528 544 560 576 592 608
624 640 656 672 688 704 720 736 752 768 784 800 816 832 848 864 880 896
912 928 944 960 976 992 1008 1024 1040 1056 1072 1088 1104 1120 1136 1152
1168 1184 1200 1216 1232 1248 1264 1280 1296 1312 1328 1344 1360 1376 1392
1408 1424 1440 1456 1472 1488 1504 1520 1536 1552 1568 1584 1600 1616 1632
1648 1664 1680 1696 1712 1728 1744 1760 1776 1792 1808 1824 1840 1856 1872
1888 1904 1920 1936 1952 1968 1984 2000 2016 2032

Every time it crashes the current process is pagedaemon, and the
instruction pointer points to 'thread_fini'.

kgdb shows:

host1# kgdb /usr/obj/usr/src/sys/LOCAL/kernel.debug
/usr/tmp/crash/vmcore.6
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
This GDB was configured as "amd64-marcel-freebsd".
#0  doadump () at pcpu.h:167
167     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) where
#0  doadump () at pcpu.h:167
#1  0x0000000000000000 in ?? ()
#2  0xffffffff802a2bd7 in boot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:410
#3  0xffffffff802a340f in panic (fmt=0xffffff007b78c500 "\uffff\022y{") at
/usr/src/sys/kern/kern_shutdown.c:566
#4  0xffffffff80412f8a in trap_fatal (frame=0xffffff007b78c500,
eva=18446742976269456104)
    at /usr/src/sys/amd64/amd64/trap.c:639
#5  0xffffffff804132af in trap_pfault (frame=0xffffffffb1c169b0,
usermode=0) at /usr/src/sys/amd64/amd64/trap.c:562
#6  0xffffffff80413553 in trap (frame=
      {tf_rdi = -1097427386128, tf_rsi = -1097440115456, tf_rdx = 99343,
tf_rcx = 0, tf_r8 = 0, tf_r9 = 0, tf_rax = 99343, tf_rbx = 0, tf_rbp =
-1097975672832, tf_r10 = 4503599627366400, tf_r11 = 3424, tf_r12 = 4,
tf_r13 = 4, tf_r14 = -1098264600680, tf_r15 = -1097364252144, tf_trapno =
12, tf_addr = 136, tf_flags = -1098264600680, tf_err = 0, tf_rip =
-2144669478, tf_cs = 8, tf_rflags = 66050, tf_rsp = -1312724368, tf_ss =
16})
    at /usr/src/sys/amd64/amd64/trap.c:341
#7  0xffffffff80400c0b in calltrap () at
/usr/src/sys/amd64/amd64/exception.S:171
#8  0xffffff007c3b00f0 in ?? ()
#9  0xffffff007b78c500 in ?? ()
#10 0x000000000001840f in ?? ()
#11 0x0000000000000000 in ?? ()
#12 0x0000000000000000 in ?? ()
#13 0x0000000000000000 in ?? ()
#14 0x000000000001840f in ?? ()
#15 0x0000000000000000 in ?? ()
#16 0xffffff005b8cd000 in ?? ()
#17 0x000ffffffffff000 in ?? ()
#18 0x0000000000000d60 in ?? ()
#19 0x0000000000000004 in ?? ()
#20 0x0000000000000004 in ?? ()
#21 0xffffff004a541f98 in ?? ()
#22 0xffffff007ffe5a10 in ?? ()
#23 0x000000000000000c in ?? ()
#24 0x0000000000000088 in ?? ()
#25 0xffffff004a541f98 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0xffffffff802af0da in thread_fini (mem=0x0, size=0) at
/usr/src/sys/kern/kern_thread.c:271
#28 0x0000000000000010 in ?? ()
#29 0x0000000000000001 in ?? ()
#30 0xffffff007ffe5a00 in ?? ()
#31 0xffffff005b8cdf98 in ?? ()
#32 0xffffffff803f67ff in zone_drain (zone=0x8) at
/usr/src/sys/vm/uma_core.c:749
#33 0xffffffff803f43b6 in zone_foreach (zfunc=0xffffffff803f6630
<zone_drain>) at /usr/src/sys/vm/uma_core.c:1494
#34 0xffffffff803f7fc9 in uma_reclaim () at
/usr/src/sys/vm/uma_core.c:2623
#35 0xffffffff803f1dac in vm_pageout () at
/usr/src/sys/vm/vm_pageout.c:674
#36 0xffffffff802898cc in fork_exit (callout=0xffffffff803f17b0
<vm_pageout>, arg=0x0, frame=0xffffffffb1c16c50)
    at /usr/src/sys/kern/kern_fork.c:791
#37 0xffffffff80400e0e in fork_trampoline () at
/usr/src/sys/amd64/amd64/exception.S:296
#38 0x0000000000000000 in ?? ()
#39 0x0000000000000000 in ?? ()
#40 0x0000000000000001 in ?? ()
#41 0x0000000000000000 in ?? ()
#42 0x0000000000000000 in ?? ()
#43 0x0000000000000000 in ?? ()
#44 0x0000000000000000 in ?? ()
#45 0x0000000000000000 in ?? ()
#46 0x0000000000000000 in ?? ()
#47 0x0000000000000000 in ?? ()
#48 0x0000000000000000 in ?? ()
#49 0x0000000000000000 in ?? ()
#50 0x0000000000000000 in ?? ()
#51 0x0000000000000000 in ?? ()
#52 0x0000000000000000 in ?? ()
#53 0x0000000000000000 in ?? ()
#54 0x0000000000000000 in ?? ()
#55 0x0000000000000000 in ?? ()
#56 0x0000000000000000 in ?? ()
#57 0x0000000000000000 in ?? ()
#58 0x0000000000000000 in ?? ()
#59 0x0000000000000000 in ?? ()
#60 0x0000000000000000 in ?? ()
#61 0x0000000000000000 in ?? ()
#62 0x0000000000000000 in ?? ()
#63 0x0000000000000000 in ?? ()
#64 0x0000000000000000 in ?? ()
#65 0x0000000000000000 in ?? ()
#66 0x0000000000000000 in ?? ()
#67 0x0000000000000000 in ?? ()
#68 0x0000000000000000 in ?? ()
#69 0x0000000000000000 in ?? ()
#70 0x00000000007ff000 in ?? ()
#71 0xffffffff8062fad4 in vm_page_max_wired ()
#72 0xffffffff8062f1c0 in vm_page_queue_free_mtx ()
#73 0x0000000000000001 in ?? ()
#74 0xffffff007b7912e8 in ?? ()
#75 0xffffff007b7d5000 in ?? ()
#76 0xffffffffb1c16a08 in ?? ()
#77 0xffffff007b78c500 in ?? ()
#78 0xffffffff802b777c in sched_switch (td=0x0, newtd=0x0, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:881
#79 0x0000000000000000 in ?? ()
#80 0x0000000000000000 in ?? ()
#81 0x0000000000000000 in ?? ()
#82 0x0000000000000000 in ?? ()
#83 0x0000000000000000 in ?? ()
#84 0x0000000000000000 in ?? ()
#85 0x0000000000000000 in ?? ()
#86 0x0000000000000000 in ?? ()
#87 0x0000000000000000 in ?? ()
#88 0x0000000000000000 in ?? ()
#89 0x0000000000000000 in ?? ()
#90 0x0000000000000000 in ?? ()
#91 0x0000000000000000 in ?? ()
#92 0x0000000000000000 in ?? ()
#93 0x0000000000000000 in ?? ()
#94 0x0000000000000000 in ?? ()
#95 0x0000000000000000 in ?? ()
#96 0x0000000000000000 in ?? ()
#97 0x0000000000000000 in ?? ()
#98 0x0000000000000000 in ?? ()
#99 0x0000000000000000 in ?? ()
#100 0x0000000000000000 in ?? ()
#101 0x0000000000000000 in ?? ()
#102 0x0000000000000000 in ?? ()
#103 0x0000000000000000 in ?? ()
#104 0x0000000000000000 in ?? ()
#105 0x0000000000000000 in ?? ()
#106 0x0000000000000000 in ?? ()
#107 0x0000000000000000 in ?? ()
#108 0x0000000000000000 in ?? ()
#109 0x0000000000000000 in ?? ()
#110 0x0000000000000000 in ?? ()
#111 0x0000000000000000 in ?? ()
#112 0x0000000000000000 in ?? ()
#113 0x0000000000000000 in ?? ()
#114 0x0000000000000000 in ?? ()
#115 0x0000000000000000 in ?? ()
#116 0x0000000000000000 in ?? ()
#117 0x0000000000000000 in ?? ()
#118 0x0000000000000000 in ?? ()
#119 0x0000000000000000 in ?? ()
#120 0x0000000000000000 in ?? ()
#121 0x0000000000000000 in ?? ()
#122 0x0000000000000000 in ?? ()
#123 0x0000000000000000 in ?? ()
#124 0x0000000000000000 in ?? ()
#125 0x0000000000000000 in ?? ()
#126 0x0000000000000000 in ?? ()
#127 0x0000000000000000 in ?? ()
#128 0x0000000000000000 in ?? ()
#129 0x0000000000000000 in ?? ()
#130 0x0000000000000000 in ?? ()
#131 0x0000000000000000 in ?? ()
#132 0x0000000000000000 in ?? ()
#133 0x0000000000000000 in ?? ()
#134 0x0000000000000000 in ?? ()
#135 0x0000000000000000 in ?? ()
#136 0x0000000000000000 in ?? ()
#137 0x0000000000000000 in ?? ()
#138 0x0000000000000000 in ?? ()
#139 0x0000000000000000 in ?? ()
#140 0x0000000000000000 in ?? ()
#141 0x0000000000000000 in ?? ()
#142 0x0000000000000000 in ?? ()
#143 0x0000000000000000 in ?? ()
#144 0x0000000000000000 in ?? ()
#145 0x0000000000000000 in ?? ()
#146 0x0000000000000000 in ?? ()
#147 0x0000000000000000 in ?? ()
#148 0x0000000000000000 in ?? ()
#149 0x0000000000000000 in ?? ()
#150 0x0000000000000000 in ?? ()
Cannot access memory at address 0xffffffffb1c17000

kernel config:

machine         amd64
cpu             HAMMER
ident           CUSTOM

makeoptions     DEBUG=-g

options         SCHED_4BSD              # 4BSD scheduler
options         INET                    # InterNETworking
options         INET6                   # IPv6 communications protocols
options         FFS                     # Berkeley Fast Filesystem
options         SOFTUPDATES             # Enable FFS soft updates support
options         UFS_ACL                 # Support for access control lists
options         UFS_DIRHASH             # Improve performance on big
directories
options         MD_ROOT                 # MD is a potential root device
options         NFSCLIENT               # Network Filesystem Client
options         NFSSERVER               # Network Filesystem Server
options         NFS_ROOT                # NFS usable as /, requires
NFSCLIENT
options         NTFS                    # NT File System
options         MSDOSFS                 # MSDOS Filesystem
options         CD9660                  # ISO 9660 Filesystem
options         PROCFS                  # Process filesystem (requires
PSEUDOFS)
options         PSEUDOFS                # Pseudo-filesystem framework
options         GEOM_GPT                # GUID Partition Tables.
options         COMPAT_43               # Needed by COMPAT_LINUX32
options         COMPAT_IA32             # Compatible with i386 binaries
options         COMPAT_FREEBSD4         # Compatible with FreeBSD4
options         COMPAT_LINUX32          # Compatible with i386 linux
binaries
options         SCSI_DELAY=15000        # Delay (in ms) before probing
SCSI
options         KTRACE                  # ktrace(1) support
options         SYSVSHM                 # SYSV-style shared memory
options         SYSVMSG                 # SYSV-style message queues
options         SYSVSEM                 # SYSV-style semaphores
options         _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time
extensions
options         KBD_INSTALL_CDEV        # install a CDEV entry in /dev
options         AHC_REG_PRETTY_PRINT    # Print register bitfields in
debug
                                        # output.  Adds ~128k to driver.
options         AHD_REG_PRETTY_PRINT    # Print register bitfields in
debug
options         ADAPTIVE_GIANT          # Giant mutex is adaptive.

options         SMP

device          atpic           # 8259A compatability
options         LINPROCFS               # Cannot be a module yet.

device          acpi
device          isa
device          pci

device          fdc

device          ata
device          atadisk         # ATA disk drives
device          ataraid         # ATA RAID drives
device          atapicd         # ATAPI CDROM drives
device          atapifd         # ATAPI floppy drives
device          atapist         # ATAPI tape drives
options         ATA_STATIC_ID   # Static device numbering

device          ahc             # AHA2940 and onboard AIC7xxx devices
device          ahd             # AHA39320/29320 and onboard AIC79xx
devices

device          aic             # Adaptec 15[012]x SCSI adapters,
AIC-6[23]60.

device          scbus           # SCSI bus (required for SCSI)
device          ch              # SCSI media changers
device          da              # Direct Access (disks)
device          sa              # Sequential Access (tape etc)
device          cd              # CD
device          pass            # Passthrough device (direct SCSI access)
device          ses             # SCSI Environmental Services (and SAF-TE)

device          atkbdc          # AT keyboard controller
device          atkbd           # AT keyboard
device          psm             # PS/2 mouse

device          vga             # VGA video card driver
device          splash          # Splash screen and screen saver support
device          sc

device          sio             # 8250, 16[45]50 based serial ports
device          ppc
device          ppbus           # Parallel port bus (required)
device          lpt             # Printer
device          ppi             # Parallel port interface device

device          em              # Intel PRO/1000 adapter Gigabit Ethernet
Card
device          miibus          # MII bus support
device          bge             # Broadcom BCM570xx Gigabit Ethernet

device          loop            # Network loopback
device          mem             # Memory and kernel memory devices
device          io              # I/O device
device          random          # Entropy device
device          ether           # Ethernet support
device          sl              # Kernel SLIP
device          ppp             # Kernel PPP
device          tun             # Packet tunnel.
device          pty             # Pseudo-ttys (telnet etc)
device          md              # Memory "disks"
device          gif             # IPv6 and IPv4 tunneling
device          faith           # IPv6-to-IPv4 relaying (translation)

device          bpf             # Berkeley packet filter

device          uhci            # UHCI PCI->USB interface
device          ohci            # OHCI PCI->USB interface
device          usb             # USB Bus (required)
device          ugen            # Generic
device          uhid            # "Human Interface Devices"
device          ukbd            # Keyboard
device          ulpt            # Printer
device          umass           # Disks/Mass storage - Requires scbus and
da
device          ums             # Mouse

any thoughts?

thanks.

-
Rob Watt


More information about the freebsd-stable mailing list