ZFS panic: spin lock held too long
Idwer Vollering
vidwer at gmail.com
Sun Mar 30 18:43:52 UTC 2014
==== dmesg (from a cold boot) ====
$ dmesg
Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014
root at snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
CPU: AMD A8-5500 APU with Radeon(tm) HD Graphics (3194.26-MHz K8-class CPU)
Origin = "AuthenticAMD" Id = 0x610f01 Family = 0x15 Model = 0x10
Stepping = 1
Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x3e98320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C>
AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
AMD Features2=0x1ebbfff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,XOP,SKINIT,WDT,LWP,FMA4,TCE,NodeId,TBM,Topology,PCXC,PNXC>
Standard Extended Features=0x8<BMI1>
TSC: P-state invariant, performance statistics
real memory = 9110028288 (8688 MB)
avail memory = 7711780864 (7354 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <CORE COREBOOT>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 16
cpu1 (AP): APIC ID: 17
cpu2 (AP): APIC ID: 18
cpu3 (AP): APIC ID: 19
ioapic0 <Version 2.1> irqs 0-23 on motherboard
random: <Software, Yarrow> initialized
kbd1 at kbdmux0
acpi0: <CORE COREBOOT> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x818-0x81b on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff iomem
0xa0000-0xbffff,0xc0000-0xdffff,0xe0000000-0xffffffff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0x2000-0x20ff mem
0xe0000000-0xefffffff,0xf0100000-0xf013ffff at device 1.0 on pci0
vgapci0: Boot video device
hdac0: <ATI (0x9902) HDA Controller> mem 0xf0140000-0xf0143fff at
device 1.1 on pci0
hdac0: hdac_get_capabilities: Invalid corb size (0)
device_attach: hdac0 attach returned 6
xhci0: <XHCI (generic) USB 3.0 controller> mem 0xf0148000-0xf0149fff
at device 16.0 on pci0
xhci0: 32 byte context size.
usbus0 on xhci0
xhci1: <XHCI (generic) USB 3.0 controller> mem 0xf014a000-0xf014bfff
at device 16.1 on pci0
xhci1: 32 byte context size.
usbus1 on xhci1
ahci0: <AMD Hudson-2 AHCI SATA controller> port
0x2410-0x2417,0x2420-0x2423,0x2418-0x241f,0x2424-0x2427,0x2400-0x240f
mem 0xf014f000-0xf014f7ff at device 17.0 on pci0
ahci0: AHCI v1.30 with 8 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ohci0: <OHCI (generic) USB controller> mem 0xf014c000-0xf014cfff at
device 18.0 on pci0
usbus2 on ohci0
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xf014f800-0xf014f8ff
at device 18.2 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
ohci1: <OHCI (generic) USB controller> mem 0xf014d000-0xf014dfff at
device 19.0 on pci0
usbus4 on ohci1
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xf014f900-0xf014f9ff
at device 19.2 on pci0
usbus5: EHCI version 1.0
usbus5 on ehci1
pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
hdac0: <Generic (0x780d1022) HDA Controller> mem 0xf0144000-0xf0147fff
at device 20.2 on pci0
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib1: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci1: <ACPI PCI bus> on pcib1
ohci2: <OHCI (generic) USB controller> mem 0xf014e000-0xf014efff at
device 20.5 on pci0
usbus6 on ohci2
sdhci_pci0: <Generic SD HCI> mem 0xf014fa00-0xf014faff at device 20.7 on pci0
sdhci_pci0: 1 slot(s) allocated
pcib2: <ACPI PCI-PCI bridge> at device 21.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 21.1 on pci0
pcib3: failed to allocate initial I/O port window: 0x1000-0x1fff
pci3: <ACPI PCI bus> on pcib3
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> mem
0xf0004000-0xf0004fff,0xf0000000-0xf0003fff at device 0.0 on pci3
re0: Using 1 MSI-X message
re0: turning off MSI enable bit.
re0: Chip rev. 0x48000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX,
100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX,
1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master,
auto, auto-flow
re0: Ethernet address: <snip>
acpi_button0: <Power Button> on acpi0
orm0: <ISA Option ROM> at iomem 0xed800-0xeffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
acpi_throttle2: <ACPI CPU Throttling> on cpu2
acpi_throttle2: failed to attach P_CNT
device_attach: acpi_throttle2 attach returned 6
acpi_throttle3: <ACPI CPU Throttling> on cpu3
acpi_throttle3: failed to attach P_CNT
device_attach: acpi_throttle3 attach returned 6
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
hdacc0: <Realtek ALC887 HDA CODEC> at cad 0 on hdac0
hdaa0: <Realtek ALC887 Audio Function Group> at nid 1 on hdacc0
pcm0: <Realtek ALC887 (Rear Analog 7.1/2.0)> at nid 20,22,21,23 and
24,26 on hdaa0
pcm1: <Realtek ALC887 (Front Analog)> at nid 27 and 25 on hdaa0
pcm2: <Realtek ALC887 (Rear Digital)> at nid 30 on hdaa0
pcm3: <Realtek ALC887 (Onboard Digital)> at nid 17 on hdaa0
random: unblocking device.
usbus0: 5.0Gbps Super Speed USB v3.0
usbus1: 5.0Gbps Super Speed USB v3.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 480Mbps High Speed USB v2.0
usbus6: 12Mbps Full Speed USB v1.0
ugen0.1: <0x1022> at usbus0
uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ugen6.1: <AMD> at usbus6
uhub1: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
ugen5.1: <AMD> at usbus5
uhub2: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
ugen4.1: <AMD> at usbus4
uhub3: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen3.1: <AMD> at usbus3
uhub4: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen2.1: <AMD> at usbus2
uhub5: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen1.1: <0x1022> at usbus1
uhub6: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <SAMSUNG HD103SJ 1AJ10001> ATA-8 SATA 2.x device
ada0: Serial Number <snip>
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <SAMSUNG HD103SJ 1AJ10001> ATA-8 SATA 2.x device
ada1: Serial Number <snip>
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad6
Netvsc initializing... SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC-low" frequency 1597128784 Hz quality 1000
uhub1: 2 ports with 2 removable, self powered
Root mount waiting for: usbus5 usbus4 usbus3 usbus2 usbus1 usbus0
uhub3: 5 ports with 5 removable, self powered
uhub5: 5 ports with 5 removable, self powered
uhub0: 4 ports with 4 removable, self powered
uhub6: 4 ports with 4 removable, self powered
Root mount waiting for: usbus5 usbus3
uhub4: 5 ports with 5 removable, self powered
uhub2: 5 ports with 5 removable, self powered
Trying to mount root from zfs:zroot_mirror_hd103sj []...
ugen3.2: <Ralink> at usbus3
run0: <1.0> on usbus3
run0: MAC/BBP RT3070 (rev 0x0201), RF RT2020 (MIMO 1T1R), address <snip>
wlan1: Ethernet address: <snip>
run0: firmware RT2870 ver. 0.236 loaded
wlan1: link state changed to UP
==== dump 1 ====
$ sudo cat info.0
Dump header from device /dev/gpt/swap0
Architecture: amd64
Architecture Version: 2
Dump Length: 2289082368B (2183 MB)
Blocksize: 512
Dumptime: Thu Mar 20 13:59:54 2014
Hostname: machete
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014
root at snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC
Panic String: spin lock held too long
Dump Parity: 422405026
Bounds: 3
Dump Status: good
$ sudo kgdb /boot/kernel/kernel.symbols vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
spin lock 0xffffffff814fa030 (smp rendezvous) held by
0xfffff8000fb42920 (tid 100428) too long
timeout stopping cpus
panic: spin lock held too long
cpuid = 3
KDB: stack backtrace:
#0 0xffffffff808e7dd0 at kdb_backtrace+0x60
#1 0xffffffff808af8b5 at panic+0x155
#2 0xffffffff8089cb71 at _mtx_lock_spin_cookie+0x241
#3 0xffffffff80c7ef54 at smp_targeted_tlb_shootdown+0xf4
#4 0xffffffff80c80335 at pmap_invalidate_page+0x265
#5 0xffffffff80c88983 at pmap_ts_referenced+0x6c3
#6 0xffffffff80b2182a at vm_pageout+0x10fa
#7 0xffffffff8088198a at fork_exit+0x9a
#8 0xffffffff80c758ce at fork_trampoline+0xe
Uptime: 1h29m59s
Dumping 2183 out of 7624 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Reading symbols from /boot/kernel/zfs.ko...done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/if_run.ko...done.
Loaded symbols for /boot/kernel/if_run.ko
Reading symbols from /boot/kernel/pf.ko...done.
Loaded symbols for /boot/kernel/pf.ko
#0 doadump (textdump=<value optimized out>) at pcpu.h:219
219 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump (textdump=<value optimized out>) at pcpu.h:219
#1 0xffffffff808af530 in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:447
#2 0xffffffff808af8f4 in panic (fmt=<value optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:754
#3 0xffffffff8089cb71 in _mtx_lock_spin_cookie (c=<value optimized
out>, tid=<value optimized out>, opts=<value optimized out>,
file=<value optimized out>,
line=<value optimized out>) at /usr/src/sys/kern/kern_mutex.c:554
#4 0xffffffff80c7ef54 in smp_targeted_tlb_shootdown (mask={__bits =
{4}}, vector=245, pmap=0xfffff8000f0bc9f8, addr1=34388402176, addr2=0)
at /usr/src/sys/amd64/amd64/mp_machdep.c:1179
#5 0xffffffff80c80335 in pmap_invalidate_page (pmap=<value optimized
out>, va=<value optimized out>) at
/usr/src/sys/amd64/amd64/pmap.c:1376
#6 0xffffffff80c88983 in pmap_ts_referenced (m=0xfffff8021168a470) at
/usr/src/sys/amd64/amd64/pmap.c:5744
#7 0xffffffff80b2182a in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1360
#8 0xffffffff8088198a in fork_exit (callout=0xffffffff80b20730
<vm_pageout>, arg=0x0, frame=0xfffffe02151b6a40) at
/usr/src/sys/kern/kern_fork.c:995
#9 0xffffffff80c758ce in fork_trampoline () at
/usr/src/sys/amd64/amd64/exception.S:606
#10 0x0000000000000000 in ?? ()
Current language: auto; currently minimal
(kgdb) up
#1 0xffffffff808af530 in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:447
447 doadump(TRUE);
(kgdb) list
442 * been completed.
443 */
444 EVENTHANDLER_INVOKE(shutdown_post_sync, howto);
445
446 if ((howto & (RB_HALT|RB_DUMP)) == RB_DUMP && !cold && !dumping)
447 doadump(TRUE);
448
449 /* Now that we're going to really halt the system... */
450 EVENTHANDLER_INVOKE(shutdown_final, howto);
451
(kgdb) up
#2 0xffffffff808af8f4 in panic (fmt=<value optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:754
754 kern_reboot(bootopt);
(kgdb) list
749 /*thread_lock(td); */
750 td->td_flags |= TDF_INPANIC;
751 /* thread_unlock(td); */
752 if (!sync_on_panic)
753 bootopt |= RB_NOSYNC;
754 kern_reboot(bootopt);
755 }
756
757 /*
758 * Support for poweroff delay.
(kgdb) up
#3 0xffffffff8089cb71 in _mtx_lock_spin_cookie (c=<value optimized
out>, tid=<value optimized out>, opts=<value optimized out>,
file=<value optimized out>,
line=<value optimized out>) at /usr/src/sys/kern/kern_mutex.c:554
554 panic("spin lock held too long");
(kgdb) list
549 printf( "spin lock %p (%s) held by %p (tid %d) too long\n",
550 m, m->lock_object.lo_name, td, td->td_tid);
551 #ifdef WITNESS
552 witness_display_spinlock(&m->lock_object, td, printf);
553 #endif
554 panic("spin lock held too long");
555 }
556
557 #ifdef SMP
558 /*
(kgdb) up
#4 0xffffffff80c7ef54 in smp_targeted_tlb_shootdown (mask={__bits =
{4}}, vector=245, pmap=0xfffff8000f0bc9f8, addr1=34388402176, addr2=0)
at /usr/src/sys/amd64/amd64/mp_machdep.c:1179
1179 mtx_lock_spin(&smp_ipi_mtx);
(kgdb) list
1174 if (CPU_EMPTY(&mask))
1175 return;
1176 }
1177 if (!(read_rflags() & PSL_I))
1178 panic("%s: interrupts disabled", __func__);
1179 mtx_lock_spin(&smp_ipi_mtx);
1180 smp_tlb_invpcid.addr = addr1;
1181 if (pmap == NULL) {
1182 smp_tlb_invpcid.pcid = 0;
1183 } else {
(kgdb) up
#5 0xffffffff80c80335 in pmap_invalidate_page (pmap=<value optimized
out>, va=<value optimized out>) at
/usr/src/sys/amd64/amd64/pmap.c:1376
1376 smp_masked_invlpg(other_cpus, pmap, va);
(kgdb) list
1371 if (pmap_pcid_enabled)
1372 CPU_AND(&other_cpus, &pmap->pm_save);
1373 else
1374 CPU_AND(&other_cpus, &pmap->pm_active);
1375 if (!CPU_EMPTY(&other_cpus))
1376 smp_masked_invlpg(other_cpus, pmap, va);
1377 }
1378 sched_unpin();
1379 }
1380
(kgdb) up
#6 0xffffffff80c88983 in pmap_ts_referenced (m=0xfffff8021168a470) at
/usr/src/sys/amd64/amd64/pmap.c:5744
5744 pmap_invalidate_page(pmap, pv->pv_va);
(kgdb) list
5739 m));
5740 pte = pmap_pde_to_pte(pde, pv->pv_va);
5741 if ((*pte & PG_A) != 0) {
5742 if (safe_to_clear_referenced(pmap, *pte)) {
5743 atomic_clear_long(pte, PG_A);
5744 pmap_invalidate_page(pmap, pv->pv_va);
5745 cleared++;
5746 } else if ((*pte & PG_W) == 0) {
5747 /*
5748 * Wired pages cannot be paged out so
(kgdb) up
#7 0xffffffff80b2182a in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1360
1360 act_delta += pmap_ts_referenced(m);
(kgdb) list
1355 * 2) The ref was transitioning to one and we saw zero.
1356 * The page lock prevents a new reference
to this page so
1357 * we need not check the reference bits.
1358 */
1359 if (m->object->ref_count != 0)
1360 act_delta += pmap_ts_referenced(m);
1361
1362 /*
1363 * Advance or decay the act_count based on recent usage.
1364 */
(kgdb) up
#8 0xffffffff8088198a in fork_exit (callout=0xffffffff80b20730
<vm_pageout>, arg=0x0, frame=0xfffffe02151b6a40) at
/usr/src/sys/kern/kern_fork.c:995
995 callout(arg, frame);
(kgdb) list
990 * cpu_set_fork_handler intercepts this function call to
991 * have this call a non-return function to stay in kernel mode.
992 * initproc has its own fork handler, but it does return.
993 */
994 KASSERT(callout != NULL, ("NULL callout in fork_exit"));
995 callout(arg, frame);
996
997 /*
998 * Check if a kernel thread misbehaved and returned
from its main
999 * function.
(kgdb) up
#9 0xffffffff80c758ce in fork_trampoline () at
/usr/src/sys/amd64/amd64/exception.S:606
606 call fork_exit
(kgdb) up
#10 0x0000000000000000 in ?? ()
(kgdb) list
601
602 ENTRY(fork_trampoline)
603 movq %r12,%rdi /* function */
604 movq %rbx,%rsi /* arg1 */
605 movq %rsp,%rdx /* trapframe pointer */
606 call fork_exit
607 MEXITCOUNT
608 jmp doreti /* Handle any ASTs */
609
610 /*
(kgdb) up
Initial frame selected; you cannot go up.
==== dump 2 ====
$ sudo cat info.1
Dump header from device /dev/gpt/swap0
Architecture: amd64
Architecture Version: 2
Dump Length: 479866880B (457 MB)
Blocksize: 512
Dumptime: Thu Mar 27 18:30:42 2014
Hostname: machete
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014
root at snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC
Panic String: spin lock held too long
Dump Parity: 287807286
Bounds: 0
Dump Status: good
$ sudo kgdb /boot/kernel/kernel.symbols vmcore.1
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
spin lock 0xffffffff814fa030 (smp rendezvous) held by
0xfffff8000e525920 (tid 100396) too long
timeout stopping cpus
panic: spin lock held too long
cpuid = 2
KDB: stack backtrace:
#0 0xffffffff808e7dd0 at kdb_backtrace+0x60
#1 0xffffffff808af8b5 at panic+0x155
#2 0xffffffff8089cb71 at _mtx_lock_spin_cookie+0x241
#3 0xffffffff80c7ef54 at smp_targeted_tlb_shootdown+0xf4
#4 0xffffffff80c80335 at pmap_invalidate_page+0x265
#5 0xffffffff80c88983 at pmap_ts_referenced+0x6c3
#6 0xffffffff80b2182a at vm_pageout+0x10fa
#7 0xffffffff8088198a at fork_exit+0x9a
#8 0xffffffff80c758ce at fork_trampoline+0xe
Uptime: 20m22s
Dumping 457 out of 7624 MB:..4%..11%..21%..32%..42%..53%..63%..74%..81%..91%
Reading symbols from /boot/kernel/zfs.ko...done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/if_run.ko...done.
Loaded symbols for /boot/kernel/if_run.ko
Reading symbols from /boot/kernel/pf.ko...done.
Loaded symbols for /boot/kernel/pf.ko
#0 doadump (textdump=<value optimized out>) at pcpu.h:219
219 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump (textdump=<value optimized out>) at pcpu.h:219
#1 0xffffffff808af530 in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:447
#2 0xffffffff808af8f4 in panic (fmt=<value optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:754
#3 0xffffffff8089cb71 in _mtx_lock_spin_cookie (c=<value optimized
out>, tid=<value optimized out>,
opts=<value optimized out>, file=<value optimized out>,
line=<value optimized out>)
at /usr/src/sys/kern/kern_mutex.c:554
#4 0xffffffff80c7ef54 in smp_targeted_tlb_shootdown (mask={__bits =
{8}}, vector=245, pmap=0xfffff8000e2542f8,
addr1=34385813504, addr2=0) at /usr/src/sys/amd64/amd64/mp_machdep.c:1179
#5 0xffffffff80c80335 in pmap_invalidate_page (pmap=<value optimized
out>, va=<value optimized out>)
at /usr/src/sys/amd64/amd64/pmap.c:1376
#6 0xffffffff80c88983 in pmap_ts_referenced (m=0xfffff80216069c60) at
/usr/src/sys/amd64/amd64/pmap.c:5744
#7 0xffffffff80b2182a in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1360
#8 0xffffffff8088198a in fork_exit (callout=0xffffffff80b20730
<vm_pageout>, arg=0x0, frame=0xfffffe02151b6a40)
at /usr/src/sys/kern/kern_fork.c:995
#9 0xffffffff80c758ce in fork_trampoline () at
/usr/src/sys/amd64/amd64/exception.S:606
#10 0x0000000000000000 in ?? ()
Current language: auto; currently minimal
(kgdb) up
#1 0xffffffff808af530 in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:447
447 doadump(TRUE);
(kgdb) list
442 * been completed.
443 */
444 EVENTHANDLER_INVOKE(shutdown_post_sync, howto);
445
446 if ((howto & (RB_HALT|RB_DUMP)) == RB_DUMP && !cold && !dumping)
447 doadump(TRUE);
448
449 /* Now that we're going to really halt the system... */
450 EVENTHANDLER_INVOKE(shutdown_final, howto);
451
(kgdb) up
#2 0xffffffff808af8f4 in panic (fmt=<value optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:754
754 kern_reboot(bootopt);
(kgdb) list
749 /*thread_lock(td); */
750 td->td_flags |= TDF_INPANIC;
751 /* thread_unlock(td); */
752 if (!sync_on_panic)
753 bootopt |= RB_NOSYNC;
754 kern_reboot(bootopt);
755 }
756
757 /*
758 * Support for poweroff delay.
(kgdb) up
#3 0xffffffff8089cb71 in _mtx_lock_spin_cookie (c=<value optimized
out>, tid=<value optimized out>,
opts=<value optimized out>, file=<value optimized out>,
line=<value optimized out>)
at /usr/src/sys/kern/kern_mutex.c:554
554 panic("spin lock held too long");
(kgdb) list
549 printf( "spin lock %p (%s) held by %p (tid %d) too long\n",
550 m, m->lock_object.lo_name, td, td->td_tid);
551 #ifdef WITNESS
552 witness_display_spinlock(&m->lock_object, td, printf);
553 #endif
554 panic("spin lock held too long");
555 }
556
557 #ifdef SMP
558 /*
(kgdb) up
#4 0xffffffff80c7ef54 in smp_targeted_tlb_shootdown (mask={__bits =
{8}}, vector=245, pmap=0xfffff8000e2542f8,
addr1=34385813504, addr2=0) at /usr/src/sys/amd64/amd64/mp_machdep.c:1179
1179 mtx_lock_spin(&smp_ipi_mtx);
(kgdb) list
1174 if (CPU_EMPTY(&mask))
1175 return;
1176 }
1177 if (!(read_rflags() & PSL_I))
1178 panic("%s: interrupts disabled", __func__);
1179 mtx_lock_spin(&smp_ipi_mtx);
1180 smp_tlb_invpcid.addr = addr1;
1181 if (pmap == NULL) {
1182 smp_tlb_invpcid.pcid = 0;
1183 } else {
(kgdb) up
#5 0xffffffff80c80335 in pmap_invalidate_page (pmap=<value optimized
out>, va=<value optimized out>)
at /usr/src/sys/amd64/amd64/pmap.c:1376
1376 smp_masked_invlpg(other_cpus, pmap, va);
(kgdb) list
1371 if (pmap_pcid_enabled)
1372 CPU_AND(&other_cpus, &pmap->pm_save);
1373 else
1374 CPU_AND(&other_cpus, &pmap->pm_active);
1375 if (!CPU_EMPTY(&other_cpus))
1376 smp_masked_invlpg(other_cpus, pmap, va);
1377 }
1378 sched_unpin();
1379 }
1380
(kgdb) up
#6 0xffffffff80c88983 in pmap_ts_referenced (m=0xfffff80216069c60) at
/usr/src/sys/amd64/amd64/pmap.c:5744
5744 pmap_invalidate_page(pmap, pv->pv_va);
(kgdb) list
5739 m));
5740 pte = pmap_pde_to_pte(pde, pv->pv_va);
5741 if ((*pte & PG_A) != 0) {
5742 if (safe_to_clear_referenced(pmap, *pte)) {
5743 atomic_clear_long(pte, PG_A);
5744 pmap_invalidate_page(pmap, pv->pv_va);
5745 cleared++;
5746 } else if ((*pte & PG_W) == 0) {
5747 /*
5748 * Wired pages cannot be paged out so
(kgdb) up
#7 0xffffffff80b2182a in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1360
1360 act_delta += pmap_ts_referenced(m);
(kgdb) list
1355 * 2) The ref was transitioning to one and we saw zero.
1356 * The page lock prevents a new reference
to this page so
1357 * we need not check the reference bits.
1358 */
1359 if (m->object->ref_count != 0)
1360 act_delta += pmap_ts_referenced(m);
1361
1362 /*
1363 * Advance or decay the act_count based on recent usage.
1364 */
(kgdb) up
#8 0xffffffff8088198a in fork_exit (callout=0xffffffff80b20730
<vm_pageout>, arg=0x0, frame=0xfffffe02151b6a40)
at /usr/src/sys/kern/kern_fork.c:995
995 callout(arg, frame);
(kgdb) list
990 * cpu_set_fork_handler intercepts this function call to
991 * have this call a non-return function to stay in kernel mode.
992 * initproc has its own fork handler, but it does return.
993 */
994 KASSERT(callout != NULL, ("NULL callout in fork_exit"));
995 callout(arg, frame);
996
997 /*
998 * Check if a kernel thread misbehaved and returned
from its main
999 * function.
(kgdb) up
#9 0xffffffff80c758ce in fork_trampoline () at
/usr/src/sys/amd64/amd64/exception.S:606
606 call fork_exit
Current language: auto; currently asm
(kgdb) list
601
602 ENTRY(fork_trampoline)
603 movq %r12,%rdi /* function */
604 movq %rbx,%rsi /* arg1 */
605 movq %rsp,%rdx /* trapframe pointer */
606 call fork_exit
607 MEXITCOUNT
608 jmp doreti /* Handle any ASTs */
609
610 /*
(kgdb) up
#10 0x0000000000000000 in ?? ()
(kgdb) list
611 * To efficiently implement classification of trap and
interrupt handlers
612 * for profiling, there must be only trap handlers between the
labels btrap
613 * and bintr, and only interrupt handlers between the labels bintr and
614 * eintr. This is implemented (partly) by including files that contain
615 * some of the handlers. Before including the files, set up a
normal asm
616 * environment so that the included files doen't need to know
that they are
617 * included.
618 */
619
620 #ifdef COMPAT_FREEBSD32
(kgdb) up
Initial frame selected; you cannot go up.
More information about the freebsd-fs
mailing list