panic possibly related to soft updates? (4.8-STABLE, Jun 12 2003)

[Ben Pfountz]

Hello list,

I have been fighting this problem for a few days now.  I have changed memory
and opened the case and monitored for heat.  I have been getting the same
panic about every 12 to 24 hours.  I can let the system sit idle, or run it
under a heavy load (cpu and disk), but the panics dont seem to be related to
system load.  It looks to me like a dangling pointer in
softdep_update_inodeblock, but I am not very experienced in this area.

The panics started recently when I was rewriting my firewall, so I suspected
a bug in IPFW.  The debug info below makes me think that the bug is related
to softupdates.

The panic is always a 'fatal trap 12'.  The fault virtual address, fault
code, code segment, processor eflags, and interupt mask is always the same.
I am not sure about the pointers.

Any suggestions?  I can run more debug commands or whatever is helpful.

Thanks in advance.

Ben

debuging information follows...

# uname -a
FreeBSD digitalpimp.princenet 4.8-STABLE FreeBSD 4.8-STABLE #1: Thu Jun 12
14:13:23 EDT 2003
root at digitalpimp.princenet:/usr/src/sys/compile/PIMPIN48  i386

panic message:
IdlePTD at phsyical address 0x0040c000
initial pcb at physical address 0x00353c40
panicstr: from debugger
panic messages:
---
Fatal trap 12: page fault while in kernel mode
fault virtual address	= 0xffff000a
fault code		= supervisor read, page not present
instruction pointer	= 0x8:0xc02639dc
stack pointer	        = 0x10:0xd7508c10
frame pointer	        = 0x10:0xd7508c10
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, def32 1, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 9579 (ksh)
interrupt mask		= bio
panic: from debugger

(kgdb) where
#0  dumpsys () at ../../kern/kern_shutdown.c:487
#1  0xc01779fb in boot (howto=260) at ../../kern/kern_shutdown.c:316
#2  0xc0177e01 in panic (fmt=0xc02dc3a4 "from debugger")
    at ../../kern/kern_shutdown.c:595
#3  0xc01362a1 in db_panic (addr=-1071236644, have_addr=0, count=-1,
    modif=0xd7508a7c "") at ../../ddb/db_command.c:435
#4  0xc0136241 in db_command (last_cmdp=0xc0328598, cmd_table=0xc03283d8,
    aux_cmd_tablep=0xc034e738) at ../../ddb/db_command.c:333
#5  0xc0136306 in db_command_loop () at ../../ddb/db_command.c:457
#6  0xc0138443 in db_trap (type=12, code=0) at ../../ddb/db_trap.c:71
#7  0xc02af742 in kdb_trap (type=12, code=0, regs=0xd7508bd0)
    at ../../i386/i386/db_interface.c:158
#8  0xc02bcce4 in trap_fatal (frame=0xd7508bd0, eva=4294901770)
    at ../../i386/i386/trap.c:969
#9  0xc02bc9bd in trap_pfault (frame=0xd7508bd0, usermode=0, eva=4294901770)
    at ../../i386/i386/trap.c:867
#10 0xc02bc563 in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, tf_edi =
0,
      tf_esi = -1035120640, tf_ebp = -682587120, tf_isp = -682587140,
      tf_ebx = -65536, tf_edx = -1033958784, tf_ecx = -65536,
      tf_eax = -1033958784, tf_trapno = 12, tf_err = 0, tf_eip
= -1071236644,
      tf_cs = 8, tf_eflags = 66071, tf_esp = -682587088, tf_ss
= -1071220514})
    at ../../i386/i386/trap.c:466
#11 0xc02639dc in worklist_remove (item=0xffff0000)
    at ../../ufs/ffs/ffs_softdep.c:467
---Type <return> to continue, or q <return> to quit---
#12 0xc02678de in softdep_update_inodeblock (ip=0xc24d5000, bp=0xcc94aeb0,
    waitfor=0) at ../../ufs/ffs/ffs_softdep.c:3847
#13 0xc026281d in ffs_update (vp=0xd73baa80, waitfor=0)
    at ../../ufs/ffs/ffs_inode.c:106
#14 0xc0262b2c in ffs_truncate (vp=0xd73baa80, length=0, flags=0,
    cred=0xc25f5c00, p=0xd7409220) at ../../ufs/ffs/ffs_inode.c:201
#15 0xc0270d52 in ufs_setattr (ap=0xd7508dfc) at
../../ufs/ufs/ufs_vnops.c:509
#16 0xc02734dd in ufs_vnoperate (ap=0xd7508dfc)
    at ../../ufs/ufs/ufs_vnops.c:2376
#17 0xc01acaba in vn_open (ndp=0xd7508ec8, fmode=1026, cmode=420)
    at vnode_if.h:305
#18 0xc01a8a2c in open (p=0xd7409220, uap=0xd7508f80)
    at ../../kern/vfs_syscalls.c:1029
#19 0xc02bcfc5 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47,
      tf_edi = -1077938272, tf_esi = -1077938292, tf_ebp = -1077938272,
      tf_isp = -682586156, tf_ebx = 134953192, tf_edx = 1537, tf_ecx = 0,
      tf_eax = 5, tf_trapno = 12, tf_err = 2, tf_eip = 134734140, tf_cs =
31,
      tf_eflags = 659, tf_esp = -1077938460, tf_ss = 47})
    at ../../i386/i386/trap.c:1175
#20 0xc02b0615 in Xint0x80_syscall ()
#21 0x8052da3 in ?? ()
#22 0x8052f87 in ?? ()
#23 0x80535a3 in ?? ()
#24 0x8053c34 in ?? ()
---Type <return> to continue, or q <return> to quit---
#25 0x8052e16 in ?? ()
#26 0x8052f87 in ?? ()
#27 0x80534c6 in ?? ()
#28 0x80534c6 in ?? ()
#29 0x805c9e1 in ?? ()
#30 0x805c59e in ?? ()
#31 0x8048141 in ?? ()

(kgdb) up 11
#11 0xc02639dc in worklist_remove (item=0xffff0000)
    at ../../ufs/ffs/ffs_softdep.c:467
467			panic("worklist_remove: lock not held");
(kgdb) p lk.lkt_held
$1 = 9579
(kgdb) p lk
$2 = {lkt_spl = 0, lkt_held = 9579}
(kgdb) p item
$3 = (struct worklist *) 0x0

(I restarted kgdb here to run 'up 12')

(kgdb) up 12
#12 0xc02678de in softdep_update_inodeblock (ip=0xc24d5000, bp=0xcc94aeb0,
    waitfor=0) at ../../ufs/ffs/ffs_softdep.c:3847
3847			WORKLIST_REMOVE(wk);
(kgdb) p wk
$1 = (struct worklist *) 0x68c460
(kgdb) p *wk
Cannot access memory at address 0x68c460.

end of debug messages.

-----
 Ben Pfountz
 B.S. Computer Science
 Computer Systems Engineer, Center for Power Electronic Systems