kernel panic on Xserve G4 -CURRENT

[Ross Gohlke]

The UP kernel ran continuously for over 48 hours with no kernel panics 
or other problems reported.

My kernel configuration looked like this:
include GENERIC
options         IPFILTER
option          IPFILTER_LOG
# SYSV For PostgreSQL
options         SHMMAXPGS=65536
options         SEMMNI=40
options         SEMMNS=240
options         SEMUME=40
options         SEMMNU=120

I replaced IPFilter firewall with PF and removed IPFilter options from 
kernel.
I moved remaining kernel options to sysctl.conf / loader.conf.
I recompiled kernel with GENERIC + options SMP.
I started getting READ_DMA errors, as reported on Oct. 26:
http://docs.freebsd.org/cgi/getmsg.cgi?fetch=0+0+archive/2008/freebsd-ppc/20081102.freebsd-ppc

Nov 16 16:16:58 rorty kernel: ad0: TIMEOUT - READ_DMA retrying (1 retry 
left) LBA=154658944

Some time last night the box went down, I'm waiting for a hard reboot now.

Next I will run UP GENERIC kernel and confirm there are no DMA errors.

Finally I will start a support group called Kernel Addicts Anonymous...

> Sorry about that.
>
> Here it is.
>
> %dmesg
> FreeBSD 8.0-CURRENT #0: Thu Nov 13 14:33:29 CST 2008
>    root at rorty.grinz.com:/usr/obj/usr/src/sys/RORTY
> WARNING: WITNESS option enabled, expect reduced performance.
> cpu0: Motorola PowerPC 7455 revision 2.1, 1000.00 MHz
> cpu0: HID0 8450c0bc<EMCP,TBEN,NAP,DPM,ICE,DCE,SGE,BTIC,LRSTK,FOLD,BHT>
> real memory  = 2135265280 (2036 MB)
> avail memory = 2067099648 (1971 MB)
>
> ...
>
> lock order reversal:
> 1st 0xc41048 user map (user map) @ /usr/src/sys/vm/vm_map.c:3115
> 2nd 0xdfd7cc ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2047
> KDB: stack backtrace:
> 0xe08cf930: at kdb_backtrace+0x4c
> 0xe08cf950: at _witness_debugger+0x3c
> 0xe08cf970: at witness_checkorder+0x8d0
> 0xe08cf9d0: at __lockmgr_args+0x23c
> 0xe08cfa50: at ffs_lock+0x9c
> 0xe08cfa80: at VOP_LOCK1_APV+0xec
> 0xe08cfaa0: at _vn_lock+0x84
> 0xe08cfaf0: at vget+0xdc
> 0xe08cfb30: at vnode_pager_lock+0x20c
> 0xe08cfb90: at vm_fault+0x218
> 0xe08cfca0: at trap_pfault+0x128
> 0xe08cfce0: at trap+0x1ac
> 0xe08cfda0: at powerpc_interrupt+0x15c
> 0xe08cfdd0: user ISI trap by 0x1818f4c: srr1=0x4000d032
>            r1=0x7fffded0 cr=0x24000048 xer=0 ctr=0
> lock order reversal:
> 1st 0xd9d16df8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2443
> 2nd 0xe38800 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:263
> KDB: stack backtrace:
> 0xe5077a20: at kdb_backtrace+0x4c
> 0xe5077a40: at _witness_debugger+0x3c
> 0xe5077a60: at witness_checkorder+0x8d0
> 0xe5077ac0: at _sx_xlock+0x90
> 0xe5077af0: at ufsdirhash_acquire+0x40
> 0xe5077b10: at ufsdirhash_remove+0x2c
> 0xe5077b40: at ufs_dirremove+0x100
> 0xe5077b80: at ufs_remove+0x8c
> 0xe5077bc0: at VOP_REMOVE_APV+0xe0
> 0xe5077be0: at kern_unlinkat+0x1b4
> 0xe5077ca0: at kern_unlink+0x24
> 0xe5077cc0: at unlink+0x1c
> 0xe5077ce0: at trap+0x460
> 0xe5077da0: at powerpc_interrupt+0x15c
> 0xe5077dd0: user SC trap by 0x21940198: srr1=0xd032
>            r1=0x7fffd570 cr=0x40000042 xer=0 ctr=0x21940190
> lock order reversal:
> 1st 0x18d716c ufs (ufs) @ /usr/src/sys/kern/vfs_mount.c:1064
> 2nd 0x18f438c devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2047
> KDB: stack backtrace:
> 0xe50727f0: at kdb_backtrace+0x4c
> 0xe5072810: at _witness_debugger+0x3c
> 0xe5072830: at witness_checkorder+0x8d0
> 0xe5072890: at __lockmgr_args+0xaf8
> 0xe5072910: at vop_stdlock+0x48
> 0xe5072930: at VOP_LOCK1_APV+0xec
> 0xe5072950: at _vn_lock+0x84
> 0xe50729a0: at vget+0xdc
> 0xe50729e0: at devfs_allocv+0x11c
> 0xe5072a20: at devfs_root+0x5c
> 0xe5072a60: at vfs_donmount+0x1458
> 0xe5072ca0: at nmount+0xd8
> 0xe5072ce0: at trap+0x460
> 0xe5072da0: at powerpc_interrupt+0x15c
> 0xe5072dd0: user SC trap by 0x218d2788: srr1=0xd032
>            r1=0x7fffcd10 cr=0x20000044 xer=0 ctr=0x218d2780
>
>
>
>
> Marcel Moolenaar wrote:
>>
>> On Nov 13, 2008, at 10:39 AM, Ross Gohlke wrote:
>>
>>> ## UP
>>> % dmesg
>>> FreeBSD 8.0-CURRENT #0: Tue Nov 11 15:01:01 CST 2008
>>>    root at rorty.grinz.com:/usr/obj/usr/src/sys/RORTY.8
>>> WARNING: WITNESS option enabled, expect reduced performance.
>>> cpu0: Motorola PowerPC 7455 revision 2.1, 1000.00 MHz
>>> cpu0: HID0 8450c0bc<EMCP,TBEN,NAP,DPM,ICE,DCE,SGE,BTIC,LRSTK,FOLD,BHT>
>>> real memory  = 2135240704 (2036 MB)
>>> avail memory = 2067062784 (1971 MB)
>>> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
>>> cpu0: dev=ff87f908 (BSP)
>>> cpu1: dev=ff880bc8
>>
>> Hmmm, this is not an UP kernel. UP kernels don't count CPUs
>> and don't announce themselves with/as FreeBSD/SMP.
>>
>> You need to remove the SMP option from your configuration
>> in order to create an UP kernel.
>>
>> FYI,
>>
>
>


-- 
Ross Gohlke > http://www.GRINZ.com
(P) 901.276.9750 (Memphis) / 310.356.6906 (Los Angeles)
(FAX/HOTLINE) 877.806.0861
AIM: grinzie GIZMO/SKYPE: rossgohlke JABBER: ross at chat.grinz.com
111 S. Highland St. #424, Memphis, TN 38111
| Always up. |