unstable 5.3 boxes

Wed Jan 5 15:29:39 PST 2005

It seems both my test and production boxes are not stable with 5.3-Release. 
I'm hoping for some insight in how to make at least the production box 
not crash.

Production Box: Dell 2600, Dual 2.8 Xeon, 2Gb ram, onboard (amr) RAID. This 
box was stable running 5.1, 5.2, and 5.2.1, but has crashed/rebooted four 
times since the upgrade to 5.3.

Test Box: Asus P2B-D, Dual PIII-650, 1Gb ram, PCI (amr) RAID. Crashed once 
since having a fresh install of 5.3. This machine has previously been stable 
running 3.x, 4.x, and 5.x until 5.3.

I was able to look at a core file after the last dump and this is what it 
shows:

(kgdb) where
#0  doadump () at pcpu.h:159
#1  0xc0561af2 in boot (howto=3067969) at /usr/src/sys/kern/kern_shutdown.c:397
#2  0xc0561ea0 in panic (fmt=0xc0728a5b "%s")   at /usr/src/sys/kern/kern_shutdown.c:553
#3  0xc06fdd71 in trap_fatal (frame=0xe93ad938, eva=0)   at /usr/src/sys/i386/i386/trap.c:809
#4  0xc06fda68 in trap_pfault (frame=0xe93ad938, usermode=0, eva=5)
     at /usr/src/sys/i386/i386/trap.c:727
#5  0xc06fd64d in trap (frame=
       {tf_fs = -382074856, tf_es = -1067974640, tf_ds = -1013972976, tf_edi = -944747424, tf_esi = 1, tf_ebp = -382019180, tf_isp = -382019228, tf_ebx = -1014786444, tf_edx = 1, tf_ecx = -1065759304, tf_eax = 31, tf_trapno = 12, tf_err = 2, tf_eip = -1067936543, tf_cs = 8, tf_eflags = 66050, tf_esp = -1065738140, tf_ss = 89}) at /usr/src/sys/i386/i386/trap.c:417
#6  0xc06ea04a in calltrap () at /usr/src/sys/i386/i386/exception.s:140
#7  0xe93a0018 in ?? ()
#8  0xc0580010 in kvprintf (fmt=0xc7b04c60 "?6w?\v6t?\v6t?", func=0x59,
     arg=0xe93ad9bc, radix=-1067875706, ap=0xc3839674 "\001")
     at /usr/src/sys/kern/subr_prf.c:643
#9  0xc058940a in selwakeuppri (sip=0x0, pri=0)
     at /usr/src/sys/kern/sys_generic.c:1096
#10 0xc0598286 in ttwakeup (tp=0xc05842e8) at /usr/src/sys/kern/tty.c:2366
#11 0xc0596c1b in ttymodem (tp=0xc3839600, flag=0)
     at /usr/src/sys/kern/tty.c:1625
#12 0xc059b245 in ptcopen (dev=0xc476e300, flag=3, devtype=8192, td=0x0)
     at linedisc.h:136
#13 0xc051ec80 in spec_open (ap=0xe93ada70)
     at /usr/src/sys/fs/specfs/spec_vnops.c:207
#14 0xc051e958 in spec_vnoperate (ap=0x0)
     at /usr/src/sys/fs/specfs/spec_vnops.c:118
#15 0xc05d06cd in vn_open_cred (ndp=0xe93adbd8, flagp=0xe93adcd8, cmode=0,
     cred=0xc74c0d00, fdidx=0) at vnode_if.h:228
#16 0xc05d027f in vn_open (ndp=0x0, flagp=0x0, cmode=0, fdidx=0)
     at /usr/src/sys/kern/vfs_vnops.c:91
#17 0xc05c93ed in kern_open (td=0xc4281c80, path=0x0, pathseg=UIO_USERSPACE,
     flags=3, mode=0) at /usr/src/sys/kern/vfs_syscalls.c:957
#18 0xc05c92e1 in open (td=0x0, uap=0x0)
     at /usr/src/sys/kern/vfs_syscalls.c:926
#19 0xc06fe105 in syscall (frame=
       {tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 134707572, tf_esi = -1, tf_ebp = -1077944968, tf_isp = -382018188, tf_ebx = 672028064, tf_edx = 672020872, tf_ecx = 674963556, tf_eax = 5, tf_trapno = 12, tf_err = 2, tf_eip = 674360387, tf_cs = 31, tf_eflags = 514, tf_esp = -1077945060, tf_ss = 47})
     at /usr/src/sys/i386/i386/trap.c:1001
#20 0xc06ea09f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:201
#21 0x0000002f in ?? ()
#22 0x0000002f in ?? ()
#23 0x0000002f in ?? ()
#24 0x08077974 in ?? ()
#25 0xffffffff in ?? ()
#26 0xbfbfdd78 in ?? ()
#27 0xe93add74 in ?? ()
#28 0x280e55a0 in ?? ()
#29 0x280e3988 in ?? ()
#30 0x283b2064 in ?? ()
#31 0x00000005 in ?? ()
#32 0x0000000c in ?? ()
#33 0x00000002 in ?? ()
#34 0x2831ec43 in ?? ()
#35 0x0000001f in ?? ()
#36 0x00000202 in ?? ()
#37 0xbfbfdd1c in ?? ()
#38 0x0000002f in ?? ()
#39 0x08049732 in ?? ()
#40 0x08049742 in ?? ()
#41 0x08049752 in ?? ()
#42 0x08049762 in ?? ()
#43 0x7d7ac000 in ?? ()
#44 0xc427e54c in ?? ()
#45 0xc4281c80 in ?? ()
#46 0xe93adaf0 in ?? ()
#47 0xe93adad8 in ?? ()
#48 0xc347c640 in ?? ()
#49 0xc0574df7 in sched_switch (td=0xffffffff, newtd=0x280e55a0, flags=Cannot access memory at address 0xbfbfdd88
)
     at /usr/src/sys/kern/sched_4bsd.c:865
Previous frame inner to this frame (corrupt stack?)
(kgdb)

I upgraded to 5.3-RELEASE-p2 after the second crash, but it didn't seem to 
change anything. I've tested both boxes with Memtest and they passed fine.

I've been using FreeBSD since 2.2.* and have never seen this kind of 
instability. Any help or advice would be much appreciated. The kernel config 
is located here:
http://kuoi.asui.uidaho.edu/~mitch/kuoi.txt
I had previously used COPTFLAGS, but I disabled them at the last compile.

The crashes have been at different times of the day and I've found no 
suspicious log activity. At the moment it's been 9 days since the last 
crash, but it has also crashed in less than a day. ACPI and HT have always 
been enabled on this box without problems.

What's the next step? Is it possible to cvsup back to 5.2.1? I've never 
downgraded a FreeBSD box.

TIA

    | Mitch Parks * mitch at kuoi.asui.uidaho.edu |

    "I bring you love and deeper understanding."
                 - Kate Bush