From nobody Mon Mar 24 10:53:47 2025 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZLqhK5v2lz5rlwl for ; Mon, 24 Mar 2025 10:55:13 +0000 (UTC) (envelope-from dirkx@webweaving.org) Received: from weser.webweaving.org (weser.webweaving.org [148.251.234.232]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (secp384r1) client-digest SHA384) (Client CN "weser.webweaving.org", Issuer "E5" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZLqhH4BtTz3mZr for ; Mon, 24 Mar 2025 10:55:11 +0000 (UTC) (envelope-from dirkx@webweaving.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=webweaving.org header.s=shared header.b="eq/ORogG"; dmarc=pass (policy=none) header.from=webweaving.org; spf=pass (mx1.freebsd.org: domain of dirkx@webweaving.org designates 148.251.234.232 as permitted sender) smtp.mailfrom=dirkx@webweaving.org Received: from smtpclient.apple (84-26-64-194.cable.dynamic.v4.ziggo.nl [84.26.64.194]) (authenticated bits=0) by weser.webweaving.org (8.18.1/8.18.1) with ESMTPSA id 52OArmDR032745 (version=TLSv1.2 cipher=ECDHE-ECDSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 24 Mar 2025 11:53:48 +0100 (CET) (envelope-from dirkx@webweaving.org) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=webweaving.org; s=shared; t=1742813628; bh=kgrbDtp6pIc2qenhPO0+t755DSexhp7Kr2Dmxtx0v20=; h=From:Subject:Date:To; b=eq/ORogGDoySHX1/IuvPfRpg5aU9tTHuztEIBIvlAaS68YSAmWZI9b2Ez2+wgvd8k 5JXaapgNm43UaRUe0jX0yPNL4eKikppBrx8eo10c3jbXYzsAl6d1NckjTXC6++fpYd 87mXOsaRTwb0C35rJuSZDUU6VD3dk8och7C+XwXU= X-Authentication-Warning: weser.webweaving.org: Host 84-26-64-194.cable.dynamic.v4.ziggo.nl [84.26.64.194] claimed to be smtpclient.apple From: Dirk-Willem van Gulik Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.400.131.1.6\)) Subject: panic: general protection fault -- ZFS scrub/checkum related Message-Id: <6A36752E-E5EB-43C8-B113-AF9240AA8F47@webweaving.org> Date: Mon, 24 Mar 2025 11:53:47 +0100 To: FreeBSD Hackers X-Mailer: Apple Mail (2.3826.400.131.1.6) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.6.4 (weser.webweaving.org [148.251.234.232]); Mon, 24 Mar 2025 11:53:48 +0100 (CET) X-Spamd-Result: default: False [2.29 / 15.00]; NEURAL_SPAM_MEDIUM(0.99)[0.987]; NEURAL_SPAM_SHORT(0.96)[0.958]; NEURAL_SPAM_LONG(0.64)[0.640]; DMARC_POLICY_ALLOW(-0.50)[webweaving.org,none]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+mx]; ONCE_RECEIVED(0.20)[]; R_DKIM_ALLOW(-0.20)[webweaving.org:s=shared]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:24940, ipnet:148.251.0.0/16, country:DE]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_ONE(0.00)[1]; RCVD_TLS_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; HAS_XAW(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[webweaving.org:+] X-Rspamd-Queue-Id: 4ZLqhH4BtTz3mZr X-Spamd-Bar: ++ On stock FreeBSD 13.4-RELEASE-p1 GENERIC amd64 -- after a dead disk = replacement in a raidz3-0 pool; after a successful re-silvering; = subsequent scrubs show one of the non-replaced disks racking up checksum = errors quite rapidly - and (each time sofar, not caught it in the actual = act yet) at around 30% and around 299 checksum errors; crash with below = kernel panic. Does this ring a bell with anyone ? Dw # tar zxf kernel-dbg.txz -C / # kgdb /usr/lib/debug/boot/kernel/kernel.debug vmcore.0 ... substantially indentical to below # kgdb /usr/lib/debug/boot/kernel/kernel.debug vmcore.1 ... substantially indentical to below # kgdb /usr/lib/debug/boot/kernel/kernel.debug vmcore.2 ... substantially indentical to below # kgdb /usr/lib/debug/boot/kernel/kernel.debug vmcore.3 GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD] .. Unread portion of the kernel message buffer: .... <7>sonewconn: pcb 0xfffff800623027c0 (148.251.234.232:993 (proto 6)): = Listen queue overflow: 49 already in queue awaiting acceptance (4 = occurrences) <118>2025-03-24T10:58:12.552431+01:00 weser.webweaving.org bandwidthd = 1338 - - IP_NUM is too low, dropping ip.... Fatal trap 9: general protection fault while in kernel mode cpuid =3D 0; apic id =3D 00 instruction pointer =3D 0x20:0xffffffff80cf2535 stack pointer =3D 0x28:0xfffffe0261056e88 frame pointer =3D 0x28:0xfffffe0261056eb0 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D resume, IOPL =3D 0 current process =3D 43896 (ps) trap number =3D 9 panic: general protection fault cpuid =3D 0 time =3D 1742810293 KDB: stack backtrace: #0 0xffffffff80c3dd05 at kdb_backtrace+0x65 #1 0xffffffff80bf1882 at vpanic+0x152 #2 0xffffffff80bf1723 at panic+0x43 #3 0xffffffff810ae069 at trap_fatal+0x389 #4 0xffffffff81085098 at calltrap+0x8 #5 0xffffffff80bd5e9c at fill_kinfo_proc+0x72c #6 0xffffffff80bd6484 at kern_proc_out+0x34 #7 0xffffffff80bd8d46 at sysctl_out_proc+0x66 #8 0xffffffff80bd6d21 at proc_iterate+0x111 #9 0xffffffff80bd8cb2 at sysctl_kern_proc+0xf2 #10 0xffffffff80c022d1 at sysctl_root_handler_locked+0x91 #11 0xffffffff80c016e4 at sysctl_root+0x244 #12 0xffffffff80c01d5e at userland_sysctl+0x15e #13 0xffffffff80c01bc0 at sys___sysctl+0x60 #14 0xffffffff810ae933 at amd64_syscall+0x103 #15 0xffffffff810859ab at fast_syscall_common+0xf8 Uptime: 14h20m32s Dumping 9139 out of 32415 = MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:53 warning: 53 /usr/src/sys/amd64/include/pcpu_aux.h: No such file or = directory (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:53 #1 doadump (textdump=3D) at = /usr/src/sys/kern/kern_shutdown.c:394 #2 0xffffffff80bf144e in kern_reboot (howto=3D260) at = /usr/src/sys/kern/kern_shutdown.c:482 #3 0xffffffff80bf18ef in vpanic (fmt=3D0xffffffff8120c09e "%s", = ap=3Dap@entry=3D0xfffffe0261056d30) at = /usr/src/sys/kern/kern_shutdown.c:921 #4 0xffffffff80bf1723 in panic (fmt=3D) at = /usr/src/sys/kern/kern_shutdown.c:845 #5 0xffffffff810ae069 in trap_fatal (frame=3D0xfffffe0261056dc0, eva=3D0)= at /usr/src/sys/amd64/amd64/trap.c:940 #6 #7 strlcpy (dst=3D, src=3D, = dsize=3Ddsize@entry=3D9) at /usr/src/sys/libkern/strlcpy.c:37 #8 0xffffffff80bd6024 in fill_kinfo_thread (td=3D0xfffff8066b02c740, = kp=3Dkp@entry=3D0xfffffe0261057220, preferthread=3Dpreferthread@entry=3D0)= at /usr/src/sys/kern/kern_proc.c:1262 #9 0xffffffff80bd5e9c in fill_kinfo_proc (p=3Dp@entry=3D0xfffffe014390757= 0, kp=3Dkp@entry=3D0xfffffe0261057220) at = /usr/src/sys/kern/kern_proc.c:1368 #10 0xffffffff80bd6484 in kern_proc_out (p=3D0xfffffe02610573bb, = p@entry=3D0xfffffe0143907570, sb=3Dsb@entry=3D0xfffffe02610576a0, = flags=3D8, flags@entry=3D1) at /usr/src/sys/kern/kern_proc.c:1551 #11 0xffffffff80bd8d46 in sysctl_out_proc (p=3D0xfffffe0143907570, = req=3D0xfffffe0261057cc8, flags=3D1) at = /usr/src/sys/kern/kern_proc.c:1594 #12 0xffffffff80bd6d21 in proc_iterate (cb=3D0xffffffff80bd8e30 = , cbarg=3Dcbarg@entry=3D0xfffffe0261057ba8) at = /usr/src/sys/kern/kern_proc.c:1617 #13 0xffffffff80bd8cb2 in sysctl_kern_proc (oidp=3D, = arg1=3D0xfffffe0261057d8c, arg2=3D, = req=3D0xfffffe0261057cc8) at /usr/src/sys/kern/kern_proc.c:1790 #14 0xffffffff80c022d1 in sysctl_root_handler_locked = (oid=3Doid@entry=3D0xffffffff81ccd0a0 , = arg1=3Darg1@entry=3D0xfffffe0261057d8c, arg2=3Darg2@entry=3D0, = req=3D0xfffffe0261057cc8, tracker=3Dtracker@entry=3D0xfffffe0261057c4= 8) at /usr/src/sys/kern/kern_sysctl.c:183 #15 0xffffffff80c016e4 in sysctl_root (oidp=3D, = arg1=3D0xfffffe0261057d8c, arg1@entry=3D0xfffffe0261057d80, arg2=3D0, = arg2@entry=3D3, req=3Dreq@entry=3D0xfffffe0261057cc8) at /usr/src/sys/kern/kern_sysctl.c:2302 #16 0xffffffff80c01d5e in userland_sysctl = (td=3Dtd@entry=3D0xfffff80433510740, name=3Dname@entry=3D0xfffffe0261057d8= 0, namelen=3D, old=3D, oldlenp=3D, inkernel=3Dinkernel@entry=3D0, new=3D0x0, newlen=3D0, = retval=3D0xfffffe0261057de8, flags=3D0) at = /usr/src/sys/kern/kern_sysctl.c:2459 #17 0xffffffff80c01bc0 in sys___sysctl (td=3D0xfffff80433510740, = uap=3D0xfffff80433510b28) at /usr/src/sys/kern/kern_sysctl.c:2332 #18 0xffffffff810ae933 in syscallenter (td=3D0xfffff80433510740) at = /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:188 #19 amd64_syscall (td=3D0xfffff80433510740, traced=3D0) at = /usr/src/sys/amd64/amd64/trap.c:1181 #20 #21 0x0000274db155238a in ?? () Backtrace stopped: Cannot access memory at address 0x274dadb15168 (kgdb) quit