Panic @r207844; current process: flowcleaner "Fatal trap 12: page fault while in kernel mode"

Wed May 12 04:27:09 UTC 2010

On Tue, May 11, 2010 at 9:24 PM, David Wolfskill <david at catwhisker.org> wrote:
> On Tue, May 11, 2010 at 08:30:09PM -0700, K. Macy wrote:
>> Please try 207949 ....
>> ...
>
> The panic (this time) didn't show up until about 10 seconds after the
> login: prompt showed up on the serial console.  Here's what it looks
> like:
>
> ...
>  3  Select option, [Enter] for default     3
>  3  or [Space] to pause timer  9           3
>  @DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDY
>
>
> GDB: no debug ports present
> KDB: debugger backends: ddb
> KDB: current backend: ddb
> Copyright (c) 1992-2010 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>        The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 9.0-CURRENT #157 r207911M: Tue May 11 20:54:25 PDT 2010
>    root at freebeast.catwhisker.org:/usr/obj/usr/src/sys/GENERIC i386
> WARNING: WITNESS option enabled, expect reduced performance.
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: Intel(R) Xeon(TM) CPU 3.60GHz (3614.55-MHz 686-class CPU)
>  Origin = "GenuineIntel"  Id = 0xf41  Family = f  Model = 4  Stepping = 1
>  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>  Features2=0x659d<SSE3,DTES64,MON,DS_CPL,EST,TM2,CNXT-ID,CX16,xTPR>
>  AMD Features=0x20100000<NX,LM>
>  TSC: P-state invariant
> real memory  = 2147483648 (2048 MB)
> avail memory = 2086129664 (1989 MB)
> ACPI APIC Table: <PTLTD          APIC  >
> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
> FreeBSD/SMP: 2 package(s) x 1 core(s)
>  cpu0 (BSP): APIC ID:  0
>  cpu1 (AP): APIC ID:  6
> ioapic0 <Version 2.0> irqs 0-23 on motherboard
> ioapic1 <Version 2.0> irqs 24-47 on motherboard
> ioapic2 <Version 2.0> irqs 48-71 on motherboard
> kbd1 at kbdmux0
> acpi0: <PTLTD   RSDT> on motherboard
> acpi0: [ITHREAD]
> acpi0: Power Button (fixed)
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
> cpu0: <ACPI CPU> on acpi0
> cpu1: <ACPI CPU> on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pci0: <ACPI PCI bus> on pcib0
> pci0: <unknown> at device 0.1 (no driver attached)
> pci0: <base peripheral> at device 1.0 (no driver attached)
> pcib1: <ACPI PCI-PCI bridge> irq 16 at device 2.0 on pci0
> pci1: <ACPI PCI bus> on pcib1
> pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
> pci2: <ACPI PCI bus> on pcib2
> aac0: <Adaptec SCSI RAID 2200S> mem 0xdc000000-0xdfffffff irq 24 at device 1.0 on pci2
> aac0: Enable Raw I/O
> aac0: New comm. interface enabled
> aac0: [ITHREAD]
> aac0: Adaptec 2200S, aac driver 2.1.9-1
> aacp0: <SCSI Passthrough Bus> on aac0
> aacp1: <SCSI Passthrough Bus> on aac0
> pcib3: <ACPI PCI-PCI bridge> at device 0.2 on pci1
> pci3: <ACPI PCI bus> on pcib3
> em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.1> port 0x2000-0x203f mem 0xd8200000-0xd821ffff irq 54 at device 2.0 on pci3
> em0: [FILTER]
> em0: Ethernet address: 00:30:48:2d:32:6a
> em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.1> port 0x2040-0x207f mem 0xd8220000-0xd823ffff irq 55 at device 2.1 on pci3
> em1: [FILTER]
> em1: Ethernet address: 00:30:48:2d:32:6b
> pcib4: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
> pci4: <ACPI PCI bus> on pcib4
> pcib5: <ACPI PCI-PCI bridge> irq 16 at device 6.0 on pci0
> pci5: <ACPI PCI bus> on pcib5
> uhci0: <Intel 82801EB (ICH5) USB controller USB-A> port 0x1400-0x141f irq 16 at device 29.0 on pci0
> uhci0: [ITHREAD]
> usbus0: <Intel 82801EB (ICH5) USB controller USB-A> on uhci0
> uhci1: <Intel 82801EB (ICH5) USB controller USB-B> port 0x1420-0x143f irq 19 at device 29.1 on pci0
> uhci1: [ITHREAD]
> usbus1: <Intel 82801EB (ICH5) USB controller USB-B> on uhci1
> uhci2: <Intel 82801EB (ICH5) USB controller USB-C> port 0x1440-0x145f irq 18 at device 29.2 on pci0
> uhci2: [ITHREAD]
> usbus2: <Intel 82801EB (ICH5) USB controller USB-C> on uhci2
> uhci3: <Intel 82801EB (ICH5) USB controller USB-D> port 0x1460-0x147f irq 16 at device 29.3 on pci0
> uhci3: [ITHREAD]
> usbus3: <Intel 82801EB (ICH5) USB controller USB-D> on uhci3
> ehci0: <Intel 82801EB/R (ICH5) USB 2.0 controller> mem 0xd8001000-0xd80013ff irq 23 at device 29.7 on pci0
> ehci0: [ITHREAD]
> usbus4: EHCI version 1.0
> usbus4: <Intel 82801EB/R (ICH5) USB 2.0 controller> on ehci0
> pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
> pci6: <ACPI PCI bus> on pcib6
> vgapci0: <VGA-compatible display> port 0x3000-0x30ff mem 0xd9000000-0xd9ffffff,0xd8300000-0xd8300fff irq 17 at device 1.0 on pci6
> isab0: <PCI-ISA bridge> at device 31.0 on pci0
> isa0: <ISA bus> on isab0
> atapci0: <Intel ICH5 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x14a0-0x14af at device 31.1 on pci0
> ata0: <ATA channel 0> on atapci0
> ata0: [ITHREAD]
> ata1: <ATA channel 1> on atapci0
> ata1: [ITHREAD]
> pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
> acpi_button0: <Power Button> on acpi0
> atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> atkbd0: [ITHREAD]
> psm0: <PS/2 Mouse> irq 12 on atkbdc0
> psm0: [GIANT-LOCKED]
> psm0: [ITHREAD]
> psm0: model Generic PS/2 mouse, device ID 0
> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
> uart0: [FILTER]
> uart0: console (9600,n,8,1)
> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
> uart1: [FILTER]
> fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
> fdc0: [FILTER]
> fd0: <1440-KB 3.5" drive> on fdc0 drive 0
> pmtimer0 on isa0
> orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff,0xc9000-0xc9fff,0xca000-0xcafff,0xcb000-0xcf7ff pnpid ORM0000 on isa0
> sc0: <System console> at flags 0x100 on isa0
> sc0: VGA <16 virtual consoles, flags=0x300>
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
> ppc0: parallel port not found.
> est0: <Enhanced SpeedStep Frequency Control> on cpu0
> est: CPU supports Enhanced Speedstep, but is not recognized.
> est: cpu_vendor GenuineIntel, msr 122d00000e24
> device_attach: est0 attach returned 6
> p4tcc0: <CPU Frequency Thermal Control> on cpu0
> est1: <Enhanced SpeedStep Frequency Control> on cpu1
> est: CPU supports Enhanced Speedstep, but is not recognized.
> est: cpu_vendor GenuineIntel, msr 122d00000e24
> device_attach: est1 attach returned 6
> p4tcc1: <CPU Frequency Thermal Control> on cpu1
> Timecounters tick every 1.000 msec
> usbus0: 12Mbps Full Speed USB v1.0
> usbus1: 12Mbps Full Speed USB v1.0
> usbus2: 12Mbps Full Speed USB v1.0
> usbus3: 12Mbps Full Speed USB v1.0
> usbus4: 480Mbps High Speed USB v2.0
> ata1: DMA limited to UDMA33, controller found non-ATA66 cable
> ugen0.1: <Intel> at usbus0
> uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
> ugen1.1: <Intel> at usbus1
> uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
> ugen2.1: <Intel> at usbus2
> uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
> ugen3.1: <Intel> at usbus3
> uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
> ugen4.1: <Intel> at usbus4
> uhub4: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4
> acd0: DVDROM <MATSHITADVD-ROM SR-8178/PZ16> at ata1-slave UDMA33
> aacd0: <RAID 1 (Mirror)> on aac0
> aacd0: 34970MB (71619584 sectors)
> aacd1: <RAID 1 (Mirror)> on aac0
> aacd1: 69974MB (143307008 sectors)
> uhub0: 2 ports with 2 removable, self powered
> uhub1: 2 ports with 2 removable, self powered
> uhub2: 2 ports with 2 removable, self powered
> uhub3: 2 ports with 2 removable, self powered
> uhub4: 8 ports with 8 removable, self powered
> ses0 at aacp0 bus 0 scbus0 target 6 lun 0
> ses0: <SUPER GEM318 0> Fixed Uninstalled SCSI-2 device
> ses0: 3.300MB/s transfers
> ses0: SAF-TE Compliant Device
> pass0 at aacp0 bus 0 scbus0 target 0 lun 0
> pass0: <SEAGATE ST336754LC 0003> Fixed Uninstalled SCSI-3 device
> pass0: 3.300MB/s transfers
> pass1 at aacp0 bus 0 scbus0 target 1 lun 0
> pass1: <SEAGATE ST336754LC 0003> Fixed Uninstalled SCSI-3 device
> pass1: 3.300MB/s transfers
> pass2 at aacp0 bus 0 scbus0 target 2 lun 0
> pass2: <SEAGATE ST373454LC 0005> Fixed Uninstalled SCSI-3 device
> pass2: 3.300MB/s transfers
> pass3 at aacp0 bus 0 scbus0 target 3 lun 0
> pass3: <SEAGATE ST373454LC 0005> Fixed Uninstalled SCSI-3 device
> pass3: 3.300MB/s transfers
> SMP: AP CPU #1 Launched!
> WARNING: WITNESS option enabled, expect reduced performance.
> Trying to mount root from ufs:/dev/aacd0s4a
> Setting hostuuid: 80f1e964-dc63-0010-89d5-0030482d326a.
> Setting hostid: 0xc74551dd.
> Entropy harvesting: interrupts ethernet point_to_point kickstart.
> Starting file system checks:
> /dev/aacd0s4a: FILE SYSTEM CLEAN; SKIPPING CHECKS
> /dev/aacd0s4a: clean, 267641 free (2161 frags, 33185 blocks, 0.3% fragmentation)
> /dev/aacd1s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS
> /dev/aacd1s1e: clean, 4751807 free (262119 frags, 561211 blocks, 2.0% fragmentation)
> /dev/aacd0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS
> /dev/aacd0s1a: clean, 422042 free (682 frags, 52670 blocks, 0.1% fragmentation)
> /dev/aacd0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS
> /dev/aacd0s1d: clean, 1076253 free (237 frags, 134502 blocks, 0.0% fragmentation)
> /dev/aacd1s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS
> /dev/aacd1s1d: clean, 11714564 free (199148 frags, 1439427 blocks, 1.2% fragmentation)
> /dev/aacd0s2a: FILE SYSTEM CLEAN; SKIPPING CHECKS
> /dev/aacd0s2a: clean, 596264 free (1144 frags, 74390 blocks, 0.2% fragmentation)
> /dev/aacd0s2d: FILE SYSTEM CLEAN; SKIPPING CHECKS
> /dev/aacd0s2d: clean, 1108504 free (24176 frags, 135541 blocks, 1.3% fragmentation)
> /dev/aacd0s3a: FILE SYSTEM CLEAN; SKIPPING CHECKS
> /dev/aacd0s3a: clean, 578034 free (1186 frags, 72106 blocks, 0.2% fragmentation)
> /dev/aacd0s3d: FILE SYSTEM CLEAN; SKIPPING CHECKS
> /dev/aacd0s3d: clean, 1070824 free (41840 frags, 128623 blocks, 2.3% fragmentation)
> /dev/aacd0s4d: FILE SYSTEM CLEAN; SKIPPING CHECKS
> /dev/aacd0s4d: clean, 1075779 free (60523 frags, 126907 blocks, 3.4% fragmentation)
> /dev/aacd0s4f: FILE SYSTEM CLEAN; SKIPPING CHECKS
> /dev/aacd0s4f: clean, 2071045 free (357 frags, 258836 blocks, 0.0% fragmentation)
> Mounting local file systems:.
> Setting hostname: freebeast.catwhisker.org.
> lock order reversal:
>  1st 0xc68406b8 ufs (ufs) @ /usr/src/sys/kern/vfs_lookup.c:502
>  2nd 0xd94fe304 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_softdep.c:11363
>  3rd 0xc6867af8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2091
> KDB: stack backtrace:
> db_trace_self_wrapper(c0cc73e5,ebe22300,c08ea0d5,c08da44b,c0cca3d8,...) at db_trace_self_wrapper+0x26
> kdb_backtrace(c08da44b,c0cca3d8,c553d030,c5540568,ebe2235c,...) at kdb_backtrace+0x29
> _witness_debugger(c0cca3d8,c6867af8,c0cbc6fd,c5540568,c0cd14f1,...) at _witness_debugger+0x25
> witness_checkorder(c6867af8,9,c0cd14f1,82b,0,...) at witness_checkorder+0x839
> __lockmgr_args(c6867af8,80100,c6867b18,0,0,...) at __lockmgr_args+0x7f9
> ffs_lock(ebe22480,c08e9e7b,c0cd0996,80100,c6867aa0,...) at ffs_lock+0x8a
> VOP_LOCK1_APV(c0dd5020,ebe22480,c638e0a4,c0defa60,c6867aa0,...) at VOP_LOCK1_APV+0xb5
> _vn_lock(c6867aa0,80100,c0cd14f1,82b,4,...) at _vn_lock+0x5e
> vget(c6867aa0,80100,c638e000,50,0,...) at vget+0xb9
> vfs_hash_get(c6848ca8,78c22,80000,c638e000,ebe225d0,...) at vfs_hash_get+0xe6
> ffs_vgetf(c6848ca8,78c22,80000,ebe225d0,1,...) at ffs_vgetf+0x49
> softdep_sync_metadata(c6840660,0,c0ced738,144,0,...) at softdep_sync_metadata+0xc92
> ffs_syncvnode(c6840660,1,c638e000,ebe22690,246,...) at ffs_syncvnode+0x3e2
> ffs_truncate(c6840660,200,0,880,c5588480,...) at ffs_truncate+0x862
> ufs_direnter(c6840660,c6867aa0,ebe22944,ebe22bd4,0,...) at ufs_direnter+0x8d4
> ufs_makeinode(ebe22bd4,0,ebe22b30,ebe22a8c,c0c03425,...) at ufs_makeinode+0x557
> ufs_create(ebe22b30,ebe22b48,0,0,ebe22ba8,...) at ufs_create+0x30
> VOP_CREATE_APV(c0dd5020,ebe22b30,ebe22bd4,ebe22ac8,0,...) at VOP_CREATE_APV+0xa5
> vn_open_cred(ebe22ba8,ebe22c5c,1a4,0,c5588480,...) at vn_open_cred+0x215
> vn_open(ebe22ba8,ebe22c5c,1a4,c637e770,0,...) at vn_open+0x3b
> kern_openat(c638e000,ffffff9c,804c5e8,0,602,...) at kern_openat+0x125
> kern_open(c638e000,804c5e8,0,601,21b6,...) at kern_open+0x35
> open(c638e000,ebe22cf8,c0cfff3b,c0caa814,c6866550,...) at open+0x30
> syscall(ebe22d38) at syscall+0x220
> Xint0x80_syscall() at Xint0x80_syscall+0x20
> --- syscall (5, FreeBSD ELF32, open), eip = 0x281744b3, esp = 0xbfbfec4c, ebp = 0xbfbfecb8 ---
> Starting Network: lo0 em0 em1.
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>        options=3<RXCSUM,TXCSUM>
>        inet 127.0.0.1 netmask 0xff000000
>        inet6 ::1 prefixlen 128
>        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
>        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>        options=289b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MAGIC>
>        ether 00:30:48:2d:32:6a
>        inet 172.16.8.10 netmask 0xffffff00 broadcast 172.16.8.255
>        inet6 fe80::230:48ff:fe2d:326a%em0 prefixlen 64 tentative scopeid 0x1
>        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>        media: Ethernet autoselect
>        status: no carrier
> em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>        options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
>        ether 00:30:48:2d:32:6b
>        media: Ethernet autoselect
>        status: no carrier
> Starting devd.
> Starting Network: em1.
> em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>        options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
>        ether 00:30:48:2d:32:6b
>        media: Ethernet autoselect
>        status: no carrier
> add net default: gateway 172.16.8.1
> add net ::ffff:0.0.0.0: gateway ::1
> add net ::0.0.0.0: gateway ::1
> add net fe80::: gateway ::1
> add net ff02::: gateway ::1
> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat
> a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout
> Creating and/or trimming log files.
> Starting syslogd.
> No core dumps found.
> Starting rpcbind.
> NFS access cache time=60
> Setting NIS domain: lmdhw.com.
> Starting ypbind.
> Starting amd.
> Clearing /tmp (X related).
> Starting mountd.
> Starting nfsd.
> Starting lpd.
> Updating motd:.
> Starting ntpd.
> Starting powerd.
> Starting rsyncd.
> Starting cvsupd.
> Configuring syscons: blanktime.
> Starting sshd.
> Starting cron.
> Starting background file system checks in 60 seconds.
>
> Tue May 11 21:20:35 PDT 2010
>
> FreeBSD/i386 (freebeast.catwhisker.org) (ttyu0)
>
> login:
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address   = 0x68
> fault code              = supervisor read, page not present
> instruction pointer     = 0x20:0xc0894306
> stack pointer           = 0x28:0xe9bc6c5c
> frame pointer           = 0x28:0xe9bc6c7c
> code segment            = base 0x0, limit 0xfffff, type 0x1b
>                        = DPL 0, pres 1, def32 1, gran 1
> processor eflags        = interrupt enabled, resume, IOPL = 0
> current process         = 20 (flowcleaner)
> [ thread pid 20 tid 100067 ]
> Stopped at      _mtx_lock_flags+0x46:   movl    0x10(%ebx),%eax
> db> bt
> Tracing pid 20 tid 100067 td 0xc5a19000
> _mtx_lock_flags(58,0,c0cd2efb,570,0,...) at _mtx_lock_flags+0x46
> flowtable_free_stale(c0e28ac0,0,c0cd2efb,600,0,...) at flowtable_free_stale+0x2fb
> flowtable_cleaner(0,e9bc6d38,c0cbf1f8,343,c63172a8,...) at flowtable_cleaner+0xd0
> fork_exit(c094fa90,0,e9bc6d38) at fork_exit+0xb8
> fork_trampoline() at fork_trampoline+0x8
> --- trap 0, eip = 0, esp = 0xe9bc6d70, ebp = 0 ---
> db>
>
>

Could you please get me the backtrace from the core?

Thanks,
Kip