[Bug 213903] Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)
bugzilla-noreply at freebsd.org
bugzilla-noreply at freebsd.org
Sun Oct 30 03:49:30 UTC 2016
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903
Bug ID: 213903
Summary: Kernel crashes from turnstile_broadcast
(/usr/src/sys/kern/subr_turnstile.c:837)
Product: Base System
Version: CURRENT
Hardware: Any
OS: Any
Status: New
Severity: Affects Only Me
Priority: ---
Component: kern
Assignee: freebsd-bugs at FreeBSD.org
Reporter: woodsb02 at freebsd.org
I am currently experiencing semi-regular kernel crashes on my FreeBSD
12-current machine. I am new to kernel debugging, and hoping someone can have a
look at the debugging output below to point me in the direction of what the
problem might be.
My machine is a FreeNAS-mini from iXsystems which I have formatted and
installed stock FreeBSD onto. My kernel is the default generic-nodebug with the
VIMAGE options added.
$ uname -a
FreeBSD freenas.woods.am 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r305311M: Sat Sep
3 12:29:01 AWST 2016
woodsb02 at freenas.woods.am:/usr/obj/usr/src/sys/GENERIC-NODEBUG-VIMAGE amd64
$ cat /usr/src/sys/amd64/conf/GENERIC-NODEBUG-VIMAGE
# SPARTICUS -- WITNESS and INVARIANTS free kernel configuration file
# for FreeBSD/amd64
include GENERIC-NODEBUG
ident GENERIC-NODEBUG-VIMAGE
#nooptions SCTP # Stream Control Transmission Protocol
options VIMAGE # VNET/Vimage support
Output from kernel crash dump debug with kgdb below:
/usr/obj/usr/src/sys/GENERIC-NODEBUG-VIMAGE)# kgdb kernel.debug
/var/crash/vmcore.last
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 04
fault virtual address = 0x30
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80b4d91c
stack pointer = 0x28:0xfffffe046813a440
frame pointer = 0x28:0xfffffe046813a470
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 33487 (sh)
Uptime: 15m16s
Dumping 1664 out of 16338 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko.debug...
done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko
Reading symbols from
/boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko...Reading symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko...Reading
symbols from
/usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko
#0 doadump (textdump=1) at pcpu.h:221
221 __asm("movq %%gs:%1,%0" : "=r" (td)
(kgdb) list *0xffffffff80b4d91c
0xffffffff80b4d91c is in turnstile_broadcast
(/usr/src/sys/kern/subr_turnstile.c:837).
832
833 /*
834 * Transfer the blocked list to the pending list.
835 */
836 mtx_lock_spin(&td_contested_lock);
837 TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue],
td_lockq);
838 mtx_unlock_spin(&td_contested_lock);
839
840 /*
841 * Give a turnstile to each thread. The last thread gets
Current language: auto; currently minimal
(kgdb) backtrace
#0 doadump (textdump=1) at pcpu.h:221
#1 0xffffffff80aea40e in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:366
#2 0xffffffff80aea9db in vpanic (fmt=<value optimized out>, ap=<value
optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3 0xffffffff80aea813 in panic (fmt=0x0) at
/usr/src/sys/kern/kern_shutdown.c:690
#4 0xffffffff8039e197 in db_panic (addr=<value optimized out>,
have_addr=false, count=0, modif=0x0) at /usr/src/sys/ddb/db_command.c:486
#5 0xffffffff8039d689 in db_command (cmd_table=<value optimized out>) at
/usr/src/sys/ddb/db_command.c:453
#6 0xffffffff8039d3e4 in db_command_loop () at
/usr/src/sys/ddb/db_command.c:506
#7 0xffffffff803a053b in db_trap (type=<value optimized out>, code=<value
optimized out>) at /usr/src/sys/ddb/db_main.c:251
#8 0xffffffff80b36b33 in kdb_trap (type=<value optimized out>, code=<value
optimized out>, tf=<value optimized out>) at /usr/src/sys/kern/subr_kdb.c:654
#9 0xffffffff80fdd441 in trap_fatal (frame=0xfffffe046813a390, eva=48) at
/usr/src/sys/amd64/amd64/trap.c:836
#10 0xffffffff80fdd673 in trap_pfault (frame=0xfffffe046813a390, usermode=0) at
/usr/src/sys/amd64/amd64/trap.c:691
#11 0xffffffff80fdcbfc in trap (frame=0xfffffe046813a390) at
/usr/src/sys/amd64/amd64/trap.c:442
#12 0xffffffff80fbf491 in calltrap () at
/usr/src/sys/amd64/amd64/exception.S:236
#13 0xffffffff80b4d91c in turnstile_broadcast (ts=0x0, queue=1) at
/usr/src/sys/kern/subr_turnstile.c:837
#14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=0xfffff803f886d960, tid=<value
optimized out>, file=<value optimized out>, line=<value optimized out>)
at /usr/src/sys/kern/kern_rwlock.c:1027
#15 0xffffffff80e525dc in vm_map_delete (map=<value optimized out>,
start=<value optimized out>, end=<value optimized out>) at
/usr/src/sys/vm/vm_map.c:2960
#16 0xffffffff80e54477 in vm_map_remove (map=0xfffff8035540f000,
start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3077
#17 0xffffffff80a9863f in exec_new_vmspace (imgp=0xfffffe046813a860,
sv=0xffffffff81a596e8) at /usr/src/sys/kern/kern_exec.c:1096
#18 0xffffffff80a6ced8 in exec_elf64_imgact (imgp=<value optimized out>) at
/usr/src/sys/kern/imgact_elf.c:896
#19 0xffffffff80a9670d in kern_execve (td=<value optimized out>, args=<value
optimized out>, mac_p=0x0) at /usr/src/sys/kern/kern_exec.c:603
#20 0xffffffff80a95b9c in sys_execve (td=0xfffff8032893aa00,
uap=0xfffffe046813ab80) at /usr/src/sys/kern/kern_exec.c:219
#21 0xffffffff80fddde8 in amd64_syscall (td=<value optimized out>, traced=0) at
subr_syscall.c:135
#22 0xffffffff80fbf77b in Xfast_syscall () at
/usr/src/sys/amd64/amd64/exception.S:396
#23 0x0000000800b468ea in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) up 11
#11 0xffffffff80fdcbfc in trap (frame=0xfffffe046813a390) at
/usr/src/sys/amd64/amd64/trap.c:442
442 (void) trap_pfault(frame, FALSE);
(kgdb) list
437
438 KASSERT(cold || td->td_ucred != NULL,
439 ("kernel trap doesn't have ucred"));
440 switch (type) {
441 case T_PAGEFLT: /* page fault */
442 (void) trap_pfault(frame, FALSE);
443 goto out;
444
445 case T_DNA:
446 if (PCB_USER_FPU(td->td_pcb))
(kgdb) print td
$1 = (struct thread *) 0xfffff8032893aa00
(kgdb) print td->td_ucred
$2 = (struct ucred *) 0xfffff8004005ec00
(kgdb) print type
$3 = 12
(kgdb) up
#12 0xffffffff80fbf491 in calltrap () at
/usr/src/sys/amd64/amd64/exception.S:236
236 call trap_check
Current language: auto; currently asm
(kgdb) up
#13 0xffffffff80b4d91c in turnstile_broadcast (ts=0x0, queue=1) at
/usr/src/sys/kern/subr_turnstile.c:837
837 TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue],
td_lockq);
Current language: auto; currently minimal
(kgdb) up
#14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=0xfffff803f886d960, tid=<value
optimized out>, file=<value optimized out>, line=<value optimized out>)
at /usr/src/sys/kern/kern_rwlock.c:1027
1027 turnstile_broadcast(ts, queue);
(kgdb) up
#15 0xffffffff80e525dc in vm_map_delete (map=<value optimized out>,
start=<value optimized out>, end=<value optimized out>) at
/usr/src/sys/vm/vm_map.c:2960
2960 VM_OBJECT_WUNLOCK(object);
(kgdb) up
#16 0xffffffff80e54477 in vm_map_remove (map=0xfffff8035540f000,
start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3077
3077 result = vm_map_delete(map, start, end);
(kgdb) up
#17 0xffffffff80a9863f in exec_new_vmspace (imgp=0xfffffe046813a860,
sv=0xffffffff81a596e8) at /usr/src/sys/kern/kern_exec.c:1096
1096 vm_map_remove(map, vm_map_min(map), vm_map_max(map));
(kgdb) up
#18 0xffffffff80a6ced8 in exec_elf64_imgact (imgp=<value optimized out>) at
/usr/src/sys/kern/imgact_elf.c:896
896 error = exec_new_vmspace(imgp, sv);
(kgdb) up
#19 0xffffffff80a9670d in kern_execve (td=<value optimized out>, args=<value
optimized out>, mac_p=0x0) at /usr/src/sys/kern/kern_exec.c:603
603 error = (*execsw[i]->ex_imgact)(imgp);
(kgdb) up
#20 0xffffffff80a95b9c in sys_execve (td=0xfffff8032893aa00,
uap=0xfffffe046813ab80) at /usr/src/sys/kern/kern_exec.c:219
219 error = kern_execve(td, &args, NULL);
(kgdb) up
#21 0xffffffff80fddde8 in amd64_syscall (td=<value optimized out>, traced=0) at
subr_syscall.c:135
135 error = (sa->callp->sy_call)(td, sa->args);
(kgdb) up
#22 0xffffffff80fbf77b in Xfast_syscall () at
/usr/src/sys/amd64/amd64/exception.S:396
396 call amd64_syscall
Current language: auto; currently asm
(kgdb) up
#23 0x0000000800b468ea in ?? ()
(kgdb) up
Initial frame selected; you cannot go up.
(kgdb) quit
After being requested for more info from Mateusz Guzik <mjguzik at gmail.com>, I
also performed the following kgdb command:
(kgdb) f 14
#14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=0xfffff803f886d960, tid=<value
optimized out>, file=<value optimized out>, line=<value optimized out>)
at /usr/src/sys/kern/kern_rwlock.c:1027
1027 turnstile_broadcast(ts, queue);
Current language: auto; currently minimal
(kgdb) x/xg c
0xfffff803f886d960: 0xfffff8032893aa00
--
You are receiving this mail because:
You are the assignee for the bug.
More information about the freebsd-bugs
mailing list