kern/176857: [panic] [suj] 9.1-RELEASE/amd64/GENERIC panic in softdepflush/remove_from_journal

Eugene Grosbein egrosbein at rdtc.ru
Mon Mar 11 16:20:02 UTC 2013 

>Number:         176857
>Category:       kern
>Synopsis:       [panic] [suj] 9.1-RELEASE/amd64/GENERIC panic in softdepflush/remove_from_journal
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Mar 11 16:20:01 UTC 2013
>Closed-Date:
>Last-Modified:
>Originator:     Eugene Grosbein
>Release:        FreeBSD 9.1-RELEASE amd64
>Organization:
RDTC JSC
>Environment:
System: FreeBSD mx.money-easy.ru 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4 09:23:10 UTC 2012 root at farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64

>Description:

	I've just installed new server located at hoster's site and hardware.
	I've used hoster's "rescue mode" running custom variant of 9.1-RELEASE/amd64.
 	I need "stock" 9.1-RELEASE, so I've downloaded FreeBSD-9.1-RELEASE-amd64-disc1.iso,
	manually partitioned disks with "gpart" and made UFS2+SUJ file systems
	using "newfs -U -j" command. Then I've extracted base and kernel there
	and created fstab and rc.conf.

	After reboot to newly installed system I've got:

# mount
/dev/ufs/root on / (ufs, local, journaled soft-updates)
devfs on /dev (devfs, local, multilabel)
/dev/ufs/usr on /usr (ufs, local, read-only)
/dev/ufs/var on /var (ufs, local, journaled soft-updates)
/dev/ufs/usrl on /usr/local (ufs, local, journaled soft-updates)
devfs on /var/named/dev (devfs, local, multilabel)

	I ran "tar xf ports.txz" and "csup stable-supfile" in parallel, both working
	with same file system /usr/local and got kernel panic soon.

	After another reboot, I've got crashdump:

Script started on Mon Mar 11 15:27:41 2013
kgdb kernel.symbols /var/crash/vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address	= 0xd0
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80b05c84
stack pointer	        = 0x28:0xffffff811256ea60
frame pointer	        = 0x28:0xffffff811256ea80
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 17 (softdepflush)
trap number		= 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80bd8240 at trap_fatal+0x290
#3 0xffffffff80bd857d at trap_pfault+0x1ed
#4 0xffffffff80bd8b9e at trap+0x3ce
#5 0xffffffff80bc315f at calltrap+0x8
#6 0xffffffff80b0b8ba at softdep_process_journal+0x3da
#7 0xffffffff80b0d1d4 at softdep_process_worklist+0x64
#8 0xffffffff80b0f817 at softdep_flush+0x197
#9 0xffffffff808bb9ef at fork_exit+0x11f
#10 0xffffffff80bc368e at fork_trampoline+0xe
Uptime: 11m53s
Dumping 662 out of 3947 MB:..3%..13%..22%..32%..42%..51%..61%..71%..83%..92%

Reading symbols from /boot/kernel/geom_cache.ko...done.
Loaded symbols for /boot/kernel/geom_cache.ko
Reading symbols from /boot/kernel/cc_chd.ko...Reading symbols from /boot/kernel/cc_chd.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/cc_chd.ko
Reading symbols from /boot/kernel/h_ertt.ko...Reading symbols from /boot/kernel/h_ertt.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/h_ertt.ko
Reading symbols from /boot/kernel/mac_portacl.ko...Reading symbols from /boot/kernel/mac_portacl.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/mac_portacl.ko
Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from /boot/kernel/ipfw.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipfw.ko
#0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
224	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) bt
#0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
#1  0xffffffff808ea3a1 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:448
#2  0xffffffff808ea897 in panic (fmt=0x1 <Address 0x1 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:636
#3  0xffffffff80bd8240 in trap_fatal (frame=0xc, eva=Variable "eva" is not available.
) at /usr/src/sys/amd64/amd64/trap.c:857
#4  0xffffffff80bd857d in trap_pfault (frame=0xffffff811256e9b0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:773
#5  0xffffffff80bd8b9e in trap (frame=0xffffff811256e9b0) at /usr/src/sys/amd64/amd64/trap.c:456
#6  0xffffffff80bc315f in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228
#7  0xffffffff80b05c84 in remove_from_journal (wk=0xfffffe0076b9f280) at /usr/src/sys/ufs/ffs/ffs_softdep.c:2711
#8  0xffffffff80b0b8ba in softdep_process_journal (mp=0xfffffe0006a89318, needwk=0x0, flags=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3234
#9  0xffffffff80b0d1d4 in softdep_process_worklist (mp=0xfffffe0006a89318, full=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:1515
#10 0xffffffff80b0f817 in softdep_flush () at /usr/src/sys/ufs/ffs/ffs_softdep.c:1373
#11 0xffffffff808bb9ef in fork_exit (callout=0xffffffff80b0f680 <softdep_flush>, arg=0x0, frame=0xffffff811256ec40) at /usr/src/sys/kern/kern_fork.c:992
#12 0xffffffff80bc368e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:602
#13 0x0000000000000000 in ?? ()
#14 0x0000000000000000 in ?? ()
#15 0x0000000000000001 in ?? ()
#16 0x0000000000000000 in ?? ()
#17 0x0000000000000000 in ?? ()
#18 0x0000000000000000 in ?? ()
#19 0x0000000000000000 in ?? ()
#20 0x0000000000000000 in ?? ()
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0xffffffff81240f80 in tdq_cpu ()
#38 0xffffffff81240f80 in tdq_cpu ()
#39 0xfffffe000697a470 in ?? ()
#40 0x0000000000000000 in ?? ()
#41 0xffffff811256e260 in ?? ()
#42 0xffffff811256e208 in ?? ()
#43 0xfffffe0002ab48e0 in ?? ()
#44 0xffffffff8091352e in sched_switch (td=0xffffffff812228a0, newtd=0x0, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/sched_ule.c:1921
Previous frame inner to this frame (corrupt stack?)
(kgdb) frame 7
#7  0xffffffff80b05c84 in remove_from_journal (wk=0xfffffe0076b9f280) at /usr/src/sys/ufs/ffs/ffs_softdep.c:2711
2711		ump = VFSTOUFS(wk->wk_mp);
(kgdb) p *(wk->wk_mp)
$1 = {mnt_mtx = {lock_object = {lo_name = 0x1 <Address 0x1 out of bounds>, lo_flags = 0, lo_data = 0, lo_witness = 0x0}, mtx_lock = 14717243392}, mnt_gen = 0, mnt_list = {
    tqe_next = 0xffffff8108c7c000, tqe_prev = 0x0}, mnt_op = 0x0, mnt_vfc = 0xffffffff808579d0, mnt_vnodecovered = 0xfffffe00069f3800, mnt_syncer = 0x0, mnt_ref = 0, mnt_nvnodelist = {
    tqh_first = 0x0, tqh_last = 0x0}, mnt_nvnodelistsize = -2128470048, mnt_activevnodelist = {tqh_first = 0x0, tqh_last = 0xfffffe000699db00}, mnt_activevnodelistsize = 111061248, 
  mnt_writeopcount = -512, mnt_kern_flag = 4096, mnt_flag = 0, mnt_pad_noasync = 1, mnt_opt = 0xfffffe00b51ff1d0, mnt_optnew = 0x2c9, mnt_maxsymlinklen = -321887340, mnt_stat = {
    f_version = 0, f_type = 0, f_flags = 0, f_bsize = 0, f_iosize = 0, f_blocks = 0, f_bfree = 514, f_bavail = 0, f_files = 0, f_ffree = 21946961920, f_syncwrites = 4096, 
    f_asyncwrites = 18446743528040235008, f_syncreads = 0, f_asyncreads = 0, f_spare = {18446744071570880064, 0, 0, 18446741874797940832, 0, 0, 18446741874797362576, 0, 
      18446741874797441920, 18446741874797360896}, f_namemax = 4096, f_owner = 0, f_fsid = {val = {4096, 0}}, 
    f_charspare = "\001\000\000\000\001\000\000\000Ðñ\031µ\000þÿÿÉ\002\000\000\000\000\000\000ÌRY®\237Ê\v\020", '\0' <repeats 40 times>, "\002\002\000\000\000\000\000", 
    f_fstypename = '\0' <repeats 15 times>, 
    f_mntfromname = "\000 ÿI\001\000\000\000\000\020\000\000\000\000\000\000\000\200I÷\200ÿÿÿ", '\0' <repeats 16 times>, "ð\217\205\200ÿÿÿÿ", '\0' <repeats 24 times>, " 1Äî\200ÿÿÿ\000\000\000\000\000\000\000", 
    f_mntonname = "  \"\201ÿÿÿÿ\000\000\000\000\000\000\000\000\200'¬\006\000þÿÿ\000¥\236\006\000þÿÿ\000\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\001\000\000\000\001", '\0' <repeats 11 times>, "¾\002\000\000\000\000\000\000þï\234&3ô¸l\000\000\000\000\000\000\000"}, mnt_cred = 0x0, mnt_data = 0x0, mnt_time = 0, mnt_iosize_max = 0, mnt_export = 0x2, 
  mnt_label = 0x0, mnt_hashseed = 0, mnt_lockref = 0, mnt_secondary_writes = -1943220224, mnt_secondary_accwrites = 1, mnt_susp_owner = 0x0, 
  mnt_gjprovider = 0xffffff8100570000 "Harness gravity with your crayon and set about creating blocks,\nramps, levers, pulleys and whatever else you fancy to get the little\nred thing to the little yellow thing.\n\nNumpty Physics is a drawing "..., mnt_explock = {lock_object = {lo_name = 0x0, lo_flags = 0, lo_data = 0, lo_witness = 0xffffffff80858ff0}, lk_lock = 0, 
    lk_exslpfail = 0, lk_timo = 0, lk_pri = 0}}
(kgdb) p (struct ufsmount *)((wk->wk_mp)->mnt_data)
$2 = (struct ufsmount *) 0x0
(kgdb) quit

Script done on Mon Mar 11 15:29:17 2013

	vmcore.0 and core.txt.0 are available for download here,
	50MB compressed: http://www.grosbein.net/crash/softdepflush/

	Now there is usage of file systems after crashdump generated
	and some ports installed:

# df -h
Filesystem       Size    Used   Avail Capacity  Mounted on
/dev/ufs/root    991M    361M    550M    40%    /
devfs            1.0k    1.0k      0B   100%    /dev
/dev/ufs/usr     495M    317M    138M    70%    /usr
/dev/ufs/var     3.9G    732M    2.9G    20%    /var
/dev/ufs/usrl    8.7G      4G    4.1G    49%    /usr/local
devfs            1.0k    1.0k      0B   100%    /var/named/dev

>How-To-Repeat:
	
	See above.
>Fix:

	Unknown.
>Release-Note:
>Audit-Trail:
>Unformatted:

More information about the freebsd-bugs mailing list