panic: general protection fault -- ZFS scrub/checkum related

From: Dirk-Willem van Gulik <dirkx_at_webweaving.org>
Date: Mon, 24 Mar 2025 10:53:47 UTC
On stock FreeBSD 13.4-RELEASE-p1 GENERIC amd64 -- after a dead disk replacement in a raidz3-0  pool; after a successful re-silvering; subsequent scrubs show one of the non-replaced disks racking up checksum errors quite rapidly - and (each time sofar, not caught it in the actual act yet) at around 30% and around 299 checksum errors; crash with below kernel panic.

Does this ring a bell with anyone ?

Dw

# tar zxf kernel-dbg.txz -C /
# kgdb /usr/lib/debug/boot/kernel/kernel.debug vmcore.0
... substantially indentical to below
# kgdb /usr/lib/debug/boot/kernel/kernel.debug vmcore.1
... substantially indentical to below
# kgdb /usr/lib/debug/boot/kernel/kernel.debug vmcore.2
... substantially indentical to below
# kgdb /usr/lib/debug/boot/kernel/kernel.debug vmcore.3
GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD]
..

Unread portion of the kernel message buffer:
....
<7>sonewconn: pcb 0xfffff800623027c0 (148.251.234.232:993 (proto 6)): Listen queue overflow: 49 already in queue awaiting acceptance (4 occurrences)
<118>2025-03-24T10:58:12.552431+01:00 weser.webweaving.org bandwidthd 1338 - - IP_NUM is too low, dropping ip....


Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer = 0x20:0xffffffff80cf2535
stack pointer         = 0x28:0xfffffe0261056e88
frame pointer         = 0x28:0xfffffe0261056eb0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 43896 (ps)
trap number = 9
panic: general protection fault
cpuid = 0
time = 1742810293
KDB: stack backtrace:
#0 0xffffffff80c3dd05 at kdb_backtrace+0x65
#1 0xffffffff80bf1882 at vpanic+0x152
#2 0xffffffff80bf1723 at panic+0x43
#3 0xffffffff810ae069 at trap_fatal+0x389
#4 0xffffffff81085098 at calltrap+0x8
#5 0xffffffff80bd5e9c at fill_kinfo_proc+0x72c
#6 0xffffffff80bd6484 at kern_proc_out+0x34
#7 0xffffffff80bd8d46 at sysctl_out_proc+0x66
#8 0xffffffff80bd6d21 at proc_iterate+0x111
#9 0xffffffff80bd8cb2 at sysctl_kern_proc+0xf2
#10 0xffffffff80c022d1 at sysctl_root_handler_locked+0x91
#11 0xffffffff80c016e4 at sysctl_root+0x244
#12 0xffffffff80c01d5e at userland_sysctl+0x15e
#13 0xffffffff80c01bc0 at sys___sysctl+0x60
#14 0xffffffff810ae933 at amd64_syscall+0x103
#15 0xffffffff810859ab at fast_syscall_common+0xf8
Uptime: 14h20m32s
Dumping 9139 out of 32415 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%


__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:53
warning: 53 /usr/src/sys/amd64/include/pcpu_aux.h: No such file or directory
(kgdb) bt
#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:53
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:394
#2  0xffffffff80bf144e in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:482
#3  0xffffffff80bf18ef in vpanic (fmt=0xffffffff8120c09e "%s", ap=ap@entry=0xfffffe0261056d30) at /usr/src/sys/kern/kern_shutdown.c:921
#4  0xffffffff80bf1723 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:845
#5  0xffffffff810ae069 in trap_fatal (frame=0xfffffe0261056dc0, eva=0) at /usr/src/sys/amd64/amd64/trap.c:940
#6  <signal handler called>
#7  strlcpy (dst=<optimized out>, src=<optimized out>, dsize=dsize@entry=9) at /usr/src/sys/libkern/strlcpy.c:37
#8  0xffffffff80bd6024 in fill_kinfo_thread (td=0xfffff8066b02c740, kp=kp@entry=0xfffffe0261057220, preferthread=preferthread@entry=0) at /usr/src/sys/kern/kern_proc.c:1262
#9  0xffffffff80bd5e9c in fill_kinfo_proc (p=p@entry=0xfffffe0143907570, kp=kp@entry=0xfffffe0261057220) at /usr/src/sys/kern/kern_proc.c:1368
#10 0xffffffff80bd6484 in kern_proc_out (p=0xfffffe02610573bb, p@entry=0xfffffe0143907570, sb=sb@entry=0xfffffe02610576a0, flags=8, flags@entry=1) at /usr/src/sys/kern/kern_proc.c:1551
#11 0xffffffff80bd8d46 in sysctl_out_proc (p=0xfffffe0143907570, req=0xfffffe0261057cc8, flags=1) at /usr/src/sys/kern/kern_proc.c:1594
#12 0xffffffff80bd6d21 in proc_iterate (cb=0xffffffff80bd8e30 <sysctl_kern_proc_iterate>, cbarg=cbarg@entry=0xfffffe0261057ba8) at /usr/src/sys/kern/kern_proc.c:1617
#13 0xffffffff80bd8cb2 in sysctl_kern_proc (oidp=<optimized out>, arg1=0xfffffe0261057d8c, arg2=<optimized out>, req=0xfffffe0261057cc8) at /usr/src/sys/kern/kern_proc.c:1790
#14 0xffffffff80c022d1 in sysctl_root_handler_locked (oid=oid@entry=0xffffffff81ccd0a0 <sysctl___kern_proc_proc>, arg1=arg1@entry=0xfffffe0261057d8c, arg2=arg2@entry=0, req=0xfffffe0261057cc8,      tracker=tracker@entry=0xfffffe0261057c48) at /usr/src/sys/kern/kern_sysctl.c:183
#15 0xffffffff80c016e4 in sysctl_root (oidp=<optimized out>, arg1=0xfffffe0261057d8c, arg1@entry=0xfffffe0261057d80, arg2=0, arg2@entry=3, req=req@entry=0xfffffe0261057cc8)
    at /usr/src/sys/kern/kern_sysctl.c:2302
#16 0xffffffff80c01d5e in userland_sysctl (td=td@entry=0xfffff80433510740, name=name@entry=0xfffffe0261057d80, namelen=<optimized out>, old=<optimized out>, oldlenp=<optimized out>,      inkernel=inkernel@entry=0, new=0x0, newlen=0, retval=0xfffffe0261057de8, flags=0) at /usr/src/sys/kern/kern_sysctl.c:2459
#17 0xffffffff80c01bc0 in sys___sysctl (td=0xfffff80433510740, uap=0xfffff80433510b28) at /usr/src/sys/kern/kern_sysctl.c:2332
#18 0xffffffff810ae933 in syscallenter (td=0xfffff80433510740) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:188
#19 amd64_syscall (td=0xfffff80433510740, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1181
#20 <signal handler called>
#21 0x0000274db155238a in ?? ()
Backtrace stopped: Cannot access memory at address 0x274dadb15168
(kgdb) quit