From nobody Thu Sep 18 11:07:27 2025 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cSCXH6SP0z68C51 for ; Thu, 18 Sep 2025 11:07:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cSCXH5Qtgz3Fb0 for ; Thu, 18 Sep 2025 11:07:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758193647; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=GiN8ljLbcX+GwhpoGToU5oje/f2251/xBs7WvNeH4X4=; b=twexN6tTq9YomfA23dzQhfHZ9MeV+ACBXqxSVMAbU+joyTOWmRDmV6QcCEOC8YXZ0yH7s1 /DQZaYGRcvwnHb/I6GS/ouMFhqkXmqnbIis0ItSexINNO/47ZQsw1ghYJ+uPe/1K5SNLj/ C0uYzU0kGVpdeEjorpqi+KDJIrX75GiIO8dcRphDid1urajRBK7bJAALmUcKONy1DyNi0g ga99qTaih1p31P8bQkFY0bxm52MD6npciS98hi0+cFfEbpHdyQ6Ze9SuqleNGxtG9Zp8Aq H+I9r5V08r+BifDrIg0T70064hxIYG0y3g3ocOdRFpSdTXjncutpibDI4r2mzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1758193647; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=GiN8ljLbcX+GwhpoGToU5oje/f2251/xBs7WvNeH4X4=; b=DTSo0y76oUZ1F0mlp5JGt+H5tr60apPAjMtWDS3mXwLkHIYZiF3DWsuA6SvnhujvZw7HW8 fUuFHoZy7xMiGtr41eCNjvTbKZLD8jcM6t+P5liu2/wVv+vmo7NYIERglrHhTK9G3KogiN CpBDgofyaQyiNLCFIriy2IEyw+E0WT2sqFkREYHSPwW3IaMdhEdIwZv/9gg/XMMcn5OSdH ySaPKIqmCnwnOvD25bZYO3wUForwCVWrbwl2k/pJrzgg6MBWeEL6zBkoOLbwxGZLn1/y5x Lm0KoBff9qYNov7MNUnHKL7e4QU1GEYwR879GaJbnzKtz4ed7iKNMbpNk7wEPA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1758193647; a=rsa-sha256; cv=none; b=JIq6M1eraPV6tjyxA2Yc1VJMKR1Jj9bRsRFe6OkjXvBgvZxA54kBpXjyIHT2hZXxJ0Rwi1 wnOLWOrt4/VoICQcCX7jVbgSSmqEkzbwaOqdshGre9ayyQG77mv/p46ZNXXwKs+Jy3b9mP TfPfe/GAm99/1a4qnZADnI0g0xdu0kaZIaifoE3ElK3yrKh31eZPUBwfjNIsHv2bKuiLXt WQ+GruKCmUm/PLbWKn9Y+wjvpnYpSzZaVaHrCK7m5KJhQSTuF6Gsf5nJ+pnzDPHcGFHzpL qaB/zQKg2H4A6B6jugUoILDdnz0PXSz+TM0GV4d8UL2HSwUiK+Bs41sIeAZPhw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4cSCXH4vbGz17mg for ; Thu, 18 Sep 2025 11:07:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 58IB7RoK099863 for ; Thu, 18 Sep 2025 11:07:27 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 58IB7RPv099862 for bugs@FreeBSD.org; Thu, 18 Sep 2025 11:07:27 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 289682] [zfs] [panic] Fatal trap 12: ZFS panic, potentially during clone, zpool problems afterwards Date: Thu, 18 Sep 2025 11:07:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.5-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: grembo@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D289682 Bug ID: 289682 Summary: [zfs] [panic] Fatal trap 12: ZFS panic, potentially during clone, zpool problems afterwards Product: Base System Version: 13.5-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: grembo@FreeBSD.org One of our machines crashed hard after upgrading to 13.5-RELEASE. It is lik= ely that ZFS clone ran at the time (machine is part of a nomad cluster that orchestrates jails by using ZFS clone - could also have been during ZFS rec= eive or while these operations were going on in parallel). No errors reported on the NVMEs, also ran long self-tests using smartctl. We did not test server memory yet. **Status zpool after the crash** (status unknown beforehand, so it's unclear if this was the cause or the ef= fect of the crash - already scrubbed the pool) # zpool status pool: zroot state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P scan: scrub repaired 0B in 00:04:44 with 0 errors on Thu Sep 18 09:56:10 = 2025 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 nvd0p3 ONLINE 0 0 1 nvd1p3 ONLINE 0 0 0 errors: No known data errors I reattach nvd0p3 and resilvered it, now it is working again. **The actual kernel panic from the crash dump** GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD] Copyright (C) 2024 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd13.5". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel... Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid =3D 0; apic id =3D 00 fault virtual address =3D 0x0 fault code =3D supervisor read data, page not present instruction pointer =3D 0x20:0xffffffff825b4d4b stack pointer =3D 0x28:0xfffffe01925adc20 frame pointer =3D 0x28:0xfffffe01925adca0 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 0 (z_wr_int_0_2) trap number =3D 12 panic: page fault cpuid =3D 0 time =3D 1758116042 KDB: stack backtrace: #0 0xffffffff80c43a35 at kdb_backtrace+0x65 #1 0xffffffff80bf7162 at vpanic+0x182 #2 0xffffffff80bf6fd3 at panic+0x43 #3 0xffffffff810b5169 at trap_fatal+0x389 #4 0xffffffff810b51b6 at trap_pfault+0x46 #5 0xffffffff8108c298 at calltrap+0x8 #6 0xffffffff825b4ef5 at abd_copy_to_buf_off+0x25 #7 0xffffffff825b6c1f at arc_buf_fill+0x9f #8 0xffffffff825bcd8f at arc_read_done+0x25f #9 0xffffffff8272b81d at zio_done+0xcbd #10 0xffffffff82725508 at zio_execute+0x38 #11 0xffffffff80c58152 at taskqueue_run_locked+0x182 #12 0xffffffff80c593a2 at taskqueue_thread_loop+0xc2 #13 0xffffffff80bb29ff at fork_exit+0x7f #14 0xffffffff8108d30e at fork_trampoline+0xe Uptime: 9m36s Dumping 4038 out of 65265 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..= 91% Reading symbols from /boot/kernel/cryptodev.ko... Reading symbols from /usr/lib/debug//boot/kernel/cryptodev.ko.debug... Reading symbols from /boot/kernel/zfs.ko... Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug... Reading symbols from /boot/kernel/geom_mirror.ko... Reading symbols from /usr/lib/debug//boot/kernel/geom_mirror.ko.debug... Reading symbols from /boot/kernel/geom_eli.ko... Reading symbols from /usr/lib/debug//boot/kernel/geom_eli.ko.debug... Reading symbols from /boot/kernel/acpi_wmi.ko... Reading symbols from /usr/lib/debug//boot/kernel/acpi_wmi.ko.debug... Reading symbols from /boot/kernel/ichsmb.ko... Reading symbols from /usr/lib/debug//boot/kernel/ichsmb.ko.debug... Reading symbols from /boot/kernel/smbus.ko... Reading symbols from /usr/lib/debug//boot/kernel/smbus.ko.debug... Reading symbols from /boot/kernel/pchtherm.ko... Reading symbols from /usr/lib/debug//boot/kernel/pchtherm.ko.debug... Reading symbols from /boot/kernel/pf.ko... Reading symbols from /usr/lib/debug//boot/kernel/pf.ko.debug... Reading symbols from /boot/kernel/if_epair.ko... Reading symbols from /usr/lib/debug//boot/kernel/if_epair.ko.debug... --Type for more, q to quit, c to continue without paging-- Reading symbols from /boot/kernel/nullfs.ko... Reading symbols from /usr/lib/debug//boot/kernel/nullfs.ko.debug... Reading symbols from /boot/kernel/if_bridge.ko... Reading symbols from /usr/lib/debug//boot/kernel/if_bridge.ko.debug... Reading symbols from /boot/kernel/bridgestp.ko... Reading symbols from /usr/lib/debug//boot/kernel/bridgestp.ko.debug... __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:53 53 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(struct pcp= u, (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:53 #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdown= .c:394 #2 0xffffffff80bf6cfe in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:482 #3 0xffffffff80bf71ba in vpanic (fmt=3D0xffffffff81210f6a "%s", ap=3Dap@entry=3D0xfffffe01925ada80) at /usr/src/sys/kern/kern_shutdown.c:921 #4 0xffffffff80bf6fd3 in panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:845 #5 0xffffffff810b5169 in trap_fatal (frame=3D0xfffffe01925adb60, eva=3D0) = at /usr/src/sys/amd64/amd64/trap.c:940 #6 0xffffffff810b51b6 in trap_pfault (frame=3D, usermode=3Dfa= lse, signo=3D, ucode=3D) at /usr/src/sys/amd64/amd64/trap.c:759 #7 #8 abd_is_gang (abd=3D0x0) at /usr/src/sys/contrib/openzfs/include/sys/abd= .h:192 #9 abd_iterate_func (abd=3D0x0, off=3Doff@entry=3D0, size=3D16384, func=3D0xffffffff825b4f00 ,=20 private=3Dprivate@entry=3D0xfffffe01925adcb8) at /usr/src/sys/contrib/openzfs/module/zfs/abd.c:805 #10 0xffffffff825b4ef5 in abd_copy_to_buf_off (buf=3D, abd= =3D0x0, off=3Doff@entry=3D0, size=3D16384) at /usr/src/sys/contrib/openzfs/module/zfs/abd.c:855 #11 0xffffffff825b6c1f in abd_copy_to_buf (buf=3D, abd=3D, size=3D) at /usr/src/sys/contrib/openzfs/include/sys/abd.h:159 #12 0xffffffff825b6c1f in arc_buf_fill (buf=3D0xfffff80603369540, spa=3D, zb=3D, flags=3D) from /boot/kernel/zfs.ko #13 0xffffffff825b894a in arc_buf_alloc_impl (hdr=3Dhdr@entry=3D0xfffff8062= ed92720, spa=3D0x0, zb=3D0xfffff804e53e8590,=20 zb@entry=3D0xfffff804e53e85a8, tag=3D, encrypted=3D0, compressed=3D4294965252, noauth=3D, fill=3D3116598480,=20 ret=3D0xfffff800082c9740) at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:2838 #14 0xffffffff825bcd8f in arc_read_done (zio=3D0xfffff804b9c384d0) at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:5790 #15 0xffffffff8272b81d in zio_done (zio=3D0xfffff804b9c384d0) at /usr/src/sys/contrib/openzfs/module/zfs/zio.c:4845 #16 0xffffffff82725508 in __zio_execute (zio=3D) at /usr/src/sys/contrib/openzfs/module/zfs/zio.c:2219 #17 zio_execute (zio=3D) at /usr/src/sys/contrib/openzfs/module/zfs/zio.c:2130 #18 0xffffffff80c58152 in taskqueue_run_locked (queue=3Dqueue@entry=3D0xfffff800082cbc00) at /usr/src/sys/kern/subr_taskqueue.c:518 #19 0xffffffff80c593a2 in taskqueue_thread_loop (arg=3Darg@entry=3D0xfffff80008334da0) at /usr/src/sys/kern/subr_taskqueue.= c:830 #20 0xffffffff80bb29ff in fork_exit (callout=3D0xffffffff80c592e0 , arg=3D0xfffff80008334da0,=20 frame=3D0xfffffe01925adf40) at /usr/src/sys/kern/kern_fork.c:1151 #21 #22 0x988b770da12f0548 in ?? () Backtrace stopped: Cannot access memory at address 0xbc8353408d0f34e0 --=20 You are receiving this mail because: You are the assignee for the bug.=