Kernel panic in zfs code; 12-STABLE

Karl Denninger karl at denninger.net
Thu Jul 18 20:13:46 UTC 2019 

FreeBSD 12.0-STABLE #2 r349024M: Thu Jun 13 18:01:16 CDT 2019    
karl at NewFS.denninger.net:/usr/obj/usr/src/amd64.amd64/sys/KSD-SMP

Note -- no patches of any sort in the ZFS code; I am NOT running any of
my former patch set.

NewFS.denninger.net dumped core - see /var/crash/vmcore.8

Thu Jul 18 15:02:54 CDT 2019

FreeBSD NewFS.denninger.net 12.0-STABLE FreeBSD 12.0-STABLE #2 r349024M:
Thu Jun 13 18:01:16 CDT 2019    
karl at NewFS.denninger.net:/usr/obj/usr/src/amd64.amd64/sys/KSD-SMP  amd64

panic: double fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:

Fatal double fault
rip 0xffffffff8271eeec rsp 0xfffffe009d4e7f60 rbp 0xfffffe009d4e8450
rax 0xfffff801b5b68000 rdx 0xababe19 rbx 0xfffff801ac399000
rcx 0x6f598 rsi 0xfffff801b5b68740 rdi 0xfffff801ac2a2668
r8 0xfffff801ac2a2668 r9 0 r10 0xfffff801ac7cf250
r11 0 r12 0xfffff801b5b685b8 r13 0xfffff801b5b68000
r14 0xfffffe0082dfb000 r15 0xfffff801b5b685b8 rflags 0x10286
cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b
fsbase 0x8002328d0 gsbase 0xffffffff8202a100 kgsbase 0
cpuid = 11; apic id = 35
panic: double fault
cpuid = 11
time = 1563479881
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfffffe000338edb0
vpanic() at vpanic+0x19d/frame 0xfffffe000338ee00
panic() at panic+0x43/frame 0xfffffe000338ee60
dblfault_handler() at dblfault_handler+0x1de/frame 0xfffffe000338ef30
Xdblfault() at Xdblfault+0xc3/frame 0xfffffe000338ef30
--- trap 0x17, rip = 0xffffffff8271eeec, rsp = 0xfffffe009d4e7f60, rbp =
0xfffffe009d4e8450 ---
vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x2c/frame
0xfffffe009d4e8450
vdev_queue_io_done() at vdev_queue_io_done+0xc8/frame 0xfffffe009d4e84a0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e84e0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8530
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8590
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e85e0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8630
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8670
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e86c0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8720
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8770
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e87c0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8800
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8850
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e88b0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8900
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8950
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8990
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e89e0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8a40
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8a90
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8ae0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8b20
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8b70
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8bd0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8c20
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8c70
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8cb0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8d00
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8d60
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8db0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8e00
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8e40
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8e90
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e8ef0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e8f40
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e8f90
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e8fd0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9020
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9080
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e90d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9120
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9160
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e91b0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9210
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9260
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e92b0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e92f0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9340
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e93a0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e93f0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9440
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9480
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e94d0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9530

zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9580
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e95d0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9610
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9660
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e96c0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9710
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9760
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e97a0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e97f0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9850
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e98a0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e98f0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9930
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9980
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e99e0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9a30
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9a80
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9ac0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9b10
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9b70
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9bc0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9c10
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9c50
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9ca0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9d00
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9d50
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9da0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9de0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9e30
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4e9e90
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9ee0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4e9f30
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4e9f70
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4e9fc0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea020
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea070
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea0c0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea100
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea150
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea1b0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea200
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea250
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea290
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea2e0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea340
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea390
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea3e0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea420

zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea470
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea4d0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea520
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea570
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea5b0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea600
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea660
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea6b0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea700
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea740
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea790
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea7f0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea840
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ea890
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ea8d0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea920
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ea980
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ea9d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eaa20
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eaa60
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eaab0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eab10
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eab60
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eabb0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eabf0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eac40
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eaca0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eacf0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ead40
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ead80
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eadd0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eae30
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eae80
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eaed0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eaf10
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eaf60
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eafc0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb010
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb060
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb0a0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb0f0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb150
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb1a0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb1f0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb230
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb280
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb2e0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb330
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb380
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb3c0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb410
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb470
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb4c0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb510
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb550
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb5a0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb600
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb650
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb6a0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb6e0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb730
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb790
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb7e0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb830
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eb870
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb8c0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eb920
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eb970
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eb9c0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eba00
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eba50
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ebab0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebb00
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ebb50
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ebb90
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebbe0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ebc40
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebc90
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ebce0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ebd20
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebd70
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ebdd0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebe20
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ebe70
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ebeb0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebf00
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ebf60
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ebfb0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec000
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec040
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec090
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec0f0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec140
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec190
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec1d0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec220
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec280
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec2d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec320
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec360
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec3b0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec410
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec460
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec4b0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec4f0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec540
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec5a0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec5f0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec640
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec680
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec6d0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec730
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec780
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec7d0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec810
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec860
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ec8c0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec910
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ec960
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ec9a0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ec9f0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4eca50
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecaa0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ecaf0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ecb30
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecb80
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ecbe0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecc30
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ecc80
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4eccc0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecd10
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ecd70
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecdc0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ece10
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ece50
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecea0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ecf00
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ecf50
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ecfa0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ecfe0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed030
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed090
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed0e0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed130
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed170
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed1c0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed220
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed270
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed2c0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed300
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed350
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed3b0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed400
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed450
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed490
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed4e0
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed540
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed590
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed5e0
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed620
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed670
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed6d0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed720
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed770
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed7b0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed800
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed860
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed8b0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4ed900
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4ed940
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4ed990
zio_vdev_io_start() at zio_vdev_io_start+0x2a7/frame 0xfffffe009d4ed9f0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4eda40
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe009d4eda90
zio_vdev_io_done() at zio_vdev_io_done+0x151/frame 0xfffffe009d4edad0
zio_execute() at zio_execute+0xac/frame 0xfffffe009d4edb20
taskqueue_run_locked() at taskqueue_run_locked+0x154/frame
0xfffffe009d4edb80
taskqueue_thread_loop() at taskqueue_thread_loop+0x98/frame
0xfffffe009d4edbb0
fork_exit() at fork_exit+0x83/frame 0xfffffe009d4edbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009d4edbf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Uptime: 15m13s
Dumping 1284 out of 12237
MB:..2%..12%..22%..32%..42%..52%..62%..72%..81%..91%

(kgdb) #0  doadump (textdump=1) at src/sys/amd64/include/pcpu.h:234
#1  0xffffffff80bdb75d in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:451
#2  0xffffffff80bdbbe9 in vpanic (fmt=<value optimized out>,
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:877
#3  0xffffffff80bdb9e3 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:804
#4  0xffffffff810d904e in dblfault_handler (frame=<value optimized out>)
    at /usr/src/sys/amd64/amd64/trap.c:1008
#5  0xffffffff810b27b3 in Xdblfault ()
    at /usr/src/sys/amd64/amd64/exception.S:294
#6  0xffffffff8271eeec in vdev_queue_io_to_issue (vq=0xfffff801b5b685b8)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:638
#7  0xffffffff8271fb48 in vdev_queue_io_done (zio=0xfffff801ac399000)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:969
#8  0xffffffff82748d91 in zio_vdev_io_done (zio=0xfffff801ac399000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3376
#9  0xffffffff82744eac in zio_execute (zio=0xfffff801ac399000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786
#10 0xffffffff82748ac7 in zio_vdev_io_start (zio=0xfffff801ac399000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3350
#11 0xffffffff82744eac in zio_execute (zio=0xfffff801ac399000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786
#12 0xffffffff8271fb95 in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:975
#13 0xffffffff82748d91 in zio_vdev_io_done (zio=0xfffff8000b376830)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3376
#14 0xffffffff82744eac in zio_execute (zio=0xfffff8000b376830)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786
#15 0xffffffff82748ac7 in zio_vdev_io_start (zio=0xfffff8000b376830)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3350
#16 0xffffffff82744eac in zio_execute (zio=0xfffff8000b376830)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786
#17 0xffffffff8271fb95 in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:975
#18 0xffffffff82748d91 in zio_vdev_io_done (zio=0xfffff801ac744830)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3376
#19 0xffffffff82744eac in zio_execute (zio=0xfffff801ac744830)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786
#20 0xffffffff82748ac7 in zio_vdev_io_start (zio=0xfffff801ac744830)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3350
#21 0xffffffff82744eac in zio_execute (zio=0xfffff801ac744830)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786
#22 0xffffffff8271fb95 in vdev_queue_io_done (zio=<value optimized out>)

.....

#275 0xffffffff82748ac7 in zio_vdev_io_start (zio=0xfffff8000b17e418)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3350
#276 0xffffffff82744eac in zio_execute (zio=0xfffff8000b17e418)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786
#277 0xffffffff8271fb95 in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:975
#278 0xffffffff82748d91 in zio_vdev_io_done (zio=0xfffff801ac355000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3376
#279 0xffffffff82744eac in zio_execute (zio=0xfffff801ac355000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786
#280 0xffffffff82748ac7 in zio_vdev_io_start (zio=0xfffff801ac355000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3350
#281 0xffffffff82744eac in zio_execute (zio=0xfffff801ac355000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786
#282 0xffffffff8271fb95 in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:975
#283 0xffffffff82748d91 in zio_vdev_io_done (zio=0xfffff8000b8b8000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3376
#284 0xffffffff82744eac in zio_execute (zio=0xfffff8000b8b8000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1786
#285 0xffffffff80c3b7f4 in taskqueue_run_locked (queue=0xfffff801a8b35100)
    at /usr/src/sys/kern/subr_taskqueue.c:467
#286 0xffffffff80c3cb28 in taskqueue_thread_loop (arg=<value optimized out>)
    at /usr/src/sys/kern/subr_taskqueue.c:773
#287 0xffffffff80b9ab23 in fork_exit (
    callout=0xffffffff80c3ca90 <taskqueue_thread_loop>,
    arg=0xfffff801a0577520, frame=0xfffffe009d4edc00)
    at /usr/src/sys/kern/kern_fork.c:1063
#288 0xffffffff810b367e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:996
#289 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb)

This is currently repeatable.  What was going on at the instant in time was:

root at NewFS:~ # /root/backup-zfs/run-backup
Begin local ZFS backup by SEND
Run backups of default [zsr/R/12.STABLE-2019-06-14 zsr/home zs/archive
zs/colo-archive zs/disk zsr/dbms/pgsql zs/work zs/dbms/ticker-9.6]
Thu Jul 18 14:57:57 CDT 2019

Import backup pool
Imported; ready to proceed
Processing zsr/R/12.STABLE-2019-06-14
Bring incremental backup up to date
attempting destroy
backup/R/12.STABLE-2019-06-14 at zfs-auto-snap_daily-2019-07-10-00h07
success
attempting destroy
backup/R/12.STABLE-2019-06-14 at zfs-auto-snap_daily-2019-07-11-00h07
success

It destroyed the snapshot on the backup volume, and panic'd immediately
thereafter.  This is an incremental send.

If I reboot the machine and re-start the backup job it will blow up when
a couple more of the incremental deletes get done.

Given the depth of the callback stack is this simply a kernel stack
exhaustion problem?  I wouldn't THINK it would be, but......

I do have this in /boot/loader.conf:

# Try to avoid kernel stack exhaustion due to TRIM storms.
kern.kstack_pages="6"

The backup volumes are spinning rust, so there should be no "TRIM"
attempts to them.  In theory.

I have the dump if someone wants me to run anything specific against it
in terms of stack frames, etc.

-- 
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4897 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20190718/008799bc/attachment.bin>

More information about the freebsd-stable mailing list