kern/107206: Background fsck causes kernel panic
Geoffrey Giesemann
geoffwa at cs.rmit.edu.au
Tue Dec 26 04:30:14 PST 2006
>Number: 107206
>Category: kern
>Synopsis: Background fsck causes kernel panic
>Confidential: no
>Severity: critical
>Priority: low
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Tue Dec 26 12:30:13 GMT 2006
>Closed-Date:
>Last-Modified:
>Originator: Geoffrey Giesemann
>Release: RELENG_6
>Organization:
>Environment:
FreeBSD iddqd.local 6.2-PRERELEASE FreeBSD 6.2-PRERELEASE #0: Sun Dec 17 02:33:32 EST 2006 geoffwa at iddqd.local:/usr/obj/usr/src/sys/IDDQD i386
>Description:
I'm running a dual P3 machine with an SMP kernel, with all the hard drives hanging off a single Areca ARC-1120 (arcmsr).
I migrated to a new larger RAID5 volume on the machine recently. Trying to do this with snapshots caused the odd (difficult-to-reproduce) kernel panic (I used dd instead). To speed up the migration I enabled write-caching to the new volumes.
Several days later, writing files to the array via Samba causes a kernel panic. Following this, the system reboots and proceeds to crash in the background fsck. This occurred twice before I booted to a 6.1 install cd, ran fsck in the foreground (which worked fine), and got everything running again. I had savecore turned on after the snapshot problems, so I now have three large coredumps.
# kgdb /usr/obj/usr/src/sys/IDDQD/kernel.debug /var/crash/vmcore.0
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
panic: softdep_setup_freeblocks: inode busy
cpuid = 0
Uptime: 4h58m59s
Dumping 1023 MB (2 chunks)
chunk 0: 1MB (159 pages) ... ok
chunk 1: 1023MB (261872 pages) 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15
#0 doadump () at pcpu.h:165
165 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) where
#0 doadump () at pcpu.h:165
#1 0xc061e541 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc061e934 in panic (fmt=0xc0905fc5 "softdep_setup_freeblocks: inode busy") at /usr/src/sys/kern/kern_shutdown.c:565
#3 0xc07bb7f9 in softdep_setup_freeblocks (ip=0xc6002c60, length=Unhandled dwarf expression opcode 0x93
) at /usr/src/sys/ufs/ffs/ffs_softdep.c:2271
#4 0xc07ae85d in ffs_truncate (vp=0xc5fff000, length=0, flags=3072, cred=0x0, td=0xc531b180) at /usr/src/sys/ufs/ffs/ffs_inode.c:278
#5 0xc07d0abe in ufs_inactive (ap=0x0) at /usr/src/sys/ufs/ufs/ufs_inode.c:126
#6 0xc08b364e in VOP_INACTIVE_APV (vop=0x0, a=0x0) at vnode_if.c:1535
#7 0xc068faf2 in vinactive (vp=0xc5fff000, td=0x0) at vnode_if.h:795
#8 0xc068f81c in vput (vp=0xc5fff000) at /usr/src/sys/kern/vfs_subr.c:2158
#9 0xc0697539 in kern_unlink (td=0xc531b180, path=0xbfbfe4b0 <Address 0xbfbfe4b0 out of bounds>, pathseg=UIO_USERSPACE)
at /usr/src/sys/kern/vfs_syscalls.c:1722
#10 0xc0697322 in unlink (td=0x0, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:1658
#11 0xc089d6f0 in syscall (frame=
{tf_fs = -1078001605, tf_es = 135528507, tf_ds = -1078001605, tf_edi = 1, tf_esi = 135534336, tf_ebp = -1077942072, tf_isp = -412762780, tf_ebx = 135461640, tf_edx = 0, tf_ecx = 5, tf_eax = 10, tf_trapno = 0, tf_err = 2, tf_eip = 674363207, tf_cs = 51, tf_eflags = 642, tf_esp = -1077944196, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:983
#12 0xc088501f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200
#13 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
# kgdb /usr/obj/usr/src/sys/IDDQD/kernel.debug /var/crash/vmcore.1
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
g_vfs_done():ufs/var[WRITE(offset=1355972608, length=16384)]error = 5
panic: initiate_write_inodeblock_ufs2: already started
cpuid = 1
Uptime: 4m18s
Dumping 1023 MB (2 chunks)
chunk 0: 1MB (159 pages) ... ok
chunk 1: 1023MB (261872 pages) 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15
#0 doadump () at pcpu.h:165
165 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) where
#0 doadump () at pcpu.h:165
#1 0xc061e541 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc061e934 in panic (fmt=0xc090659d "initiate_write_inodeblock_ufs2: already started") at /usr/src/sys/kern/kern_shutdown.c:565
#3 0xc07bf8ff in initiate_write_inodeblock_ufs2 (inodedep=0xc550ab80, bp=0x0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:4022
#4 0xc07beed0 in softdep_disk_io_initiation (bp=0xd8d1d488) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3757
#5 0xc07c9e9b in ffs_geom_strategy (bo=0xc4f2f3f0, bp=0xd8d1d488) at buf.h:433
#6 0xc0677cc9 in bufwrite (bp=0xd8d1d488) at buf.h:426
#7 0xc07c9cf8 in ffs_bufwrite (bp=0xd8d1d488) at /usr/src/sys/ufs/ffs/ffs_vfsops.c:1740
#8 0xc07ae167 in ffs_update (vp=0xc51bc440, waitfor=1) at buf.h:410
#9 0xc07ca480 in ffs_syncvnode (vp=0xc51bc440, waitfor=1) at /usr/src/sys/ufs/ffs/ffs_vnops.c:330
#10 0xc07ae760 in ffs_truncate (vp=0xc51bc440, length=8192, flags=2048, cred=0xc4afe780, td=0xc4d2d780)
at /usr/src/sys/ufs/ffs/ffs_inode.c:268
#11 0xc07d5b4d in ufs_setattr (ap=0x0) at /usr/src/sys/ufs/ufs/ufs_vnops.c:565
#12 0xc08b2c2e in VOP_SETATTR_APV (vop=0x0, a=0xe5346c14) at vnode_if.c:588
#13 0xc069a135 in ftruncate (td=0xc4d2d780, uap=0xe5346d04) at vnode_if.h:314
#14 0xc089d6f0 in syscall (frame=
{tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = 7496, tf_esi = 135566592, tf_ebp = -1077944920, tf_isp = -449548956, tf_ebx = 674493184, tf_edx = 0, tf_ecx = 246, tf_eax = 198, tf_trapno = 12, tf_err = 2, tf_eip = 674431511, tf_cs = 51, tf_eflags = 582, tf_esp = -1077944964, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:983
#15 0xc088501f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200
#16 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
# kgdb /usr/obj/usr/src/sys/IDDQD/kernel.debug /var/crash/vmcore.2
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
panic: handle_written_filepage: attached
cpuid = 0
Uptime: 5m42s
Dumping 1023 MB (2 chunks)
chunk 0: 1MB (159 pages) ... ok
chunk 1: 1023MB (261872 pages) 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15
#0 doadump () at pcpu.h:165
165 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) where
#0 doadump () at pcpu.h:165
#1 0xc061e541 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc061e934 in panic (fmt=0xc09069a0 "handle_written_filepage: attached") at /usr/src/sys/kern/kern_shutdown.c:565
#3 0xc07c0ff5 in handle_written_filepage (pagedep=0xc52bb900, bp=0xd8be3378) at /usr/src/sys/ufs/ffs/ffs_softdep.c:4772
#4 0xc07c0246 in softdep_disk_write_complete (bp=0xd8be3378) at /usr/src/sys/ufs/ffs/ffs_softdep.c:4244
#5 0xc067cec2 in bufdone (bp=0xd8be3378) at buf.h:440
#6 0xc05d57ad in g_vfs_done (bip=0x0) at /usr/src/sys/geom/geom_vfs.c:86
#7 0xc067cae9 in biodone (bp=0xc52b6738) at /usr/src/sys/kern/vfs_bio.c:2911
#8 0xc05d2cc3 in g_io_schedule_up (tp=0xc4b02000) at /usr/src/sys/geom/geom_io.c:490
#9 0xc05d2fe8 in g_up_procbody () at /usr/src/sys/geom/geom_kern.c:95
#10 0xc05fff90 in fork_exit (callout=0xc05d2f30 <g_up_procbody>, arg=0x0, frame=0x0) at /usr/src/sys/kern/kern_fork.c:821
#11 0xc088502c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208
I'm willing to dick around with the box a bit to reproduce the problem, but I'd like to avoid losing any data in the process.
>How-To-Repeat:
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
More information about the freebsd-bugs
mailing list