From nobody Sun Jan 02 17:25:37 2022 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6F3EB192A2F6 for ; Sun, 2 Jan 2022 17:25:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JRm415rwvz3pcL for ; Sun, 2 Jan 2022 17:25:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A4FA915778 for ; Sun, 2 Jan 2022 17:25:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 202HPbIT024623 for ; Sun, 2 Jan 2022 17:25:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 202HPbni024622 for bugs@FreeBSD.org; Sun, 2 Jan 2022 17:25:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 260884] [zfs] Panic in zfs_onexit_destroy Date: Sun, 02 Jan 2022 17:25:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: grembo@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1641144337; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=KHp1hm6K7wNAd6FJKiroqs0sLXA8Eg3AHNyLgUjDxv4=; b=WwXf2dilNpyDUSydTxSZJlzjdc5VrtN2NehwN2EAM4liaU/aiTJRbzX/Qds3Nkwx7epOmR 7jPaPEPUE327wvjd+yL0/Ai/GIfxgv/7pNA9eDm+DoBgWDDDwroEq+Sj73pQtI8Z/9kTze WAUwowCABzC00cM4FiA/QLec+q7QSgjsWxcZuZCwelKaQ2VDHrXhivPMQqnKMGWzOb70Ah az07Ng4ceid3R6pN7eajGa1bgaUpBzbbZon2pBkpZVjunhr9A3QSa4jR7lSVtC8/mIlOAO GoaM1ulIkd/JFZbuAErbbiYCduYUvusldjffdu0iIdyV4raz/axeIXVCEEVU5A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1641144337; a=rsa-sha256; cv=none; b=C847kKA85yVj0cf4usVToYCK2Sw2PsM1oRnxD2ZKXhH6KucSs0crauVS64F2WHQruOHtgG 91iC7GjPQiRVJbwVGO4VlsKiBK5UPqQf3gYBEj8eKXpMgltiKueXTdOP1dF9rX8j0RXHTW Fngfmg2Qx9/9nbAAI2RSL7/m8kMQIkSLIEssPmI//FD9U//8EctZz/XY0Z5HTv6qkIarRl CkCJYzhoHOS3l6zpUkJOhClow0Hr+nVS7FNMaiJjeKHCupAjfqshrhyqB27Ov75qiVdDrJ wrtsO6o4jWDL910OKXLhUtpAEGO7H2k7ytSHOJrM7tOk2/J0oAIRb1gHFJnDow== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D260884 Bug ID: 260884 Summary: [zfs] Panic in zfs_onexit_destroy Product: Base System Version: 13.0-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: grembo@FreeBSD.org I see this problem on multiple hosts running a couple of ZFS clone based ja= ils (orchestrated by nomad/pot). As pot calls `zfs list` once per second per running jail, this adds up to 10-30 calls to `zfs list` per second per node. After a few days, all hosts consistently crash with a panic, which seems to happen while calling `zfs`. This looks a lot like this bug reported in True= NAS: https://jira.ixsystems.com/browse/NAS-108891 It seems like the underlying locking problem was already fixed in OpenZFS upstream, but FreeBSD 13.0-RELEASE is using an older version. As far as I c= an see it, would be very easy to apply the fix from here to resolve a potential errata and create 13.0-RELEASE-p6 from that: https://github.com/openzfs/zfs/commit/f845b2dd1c60 You can find more context about my use case here: https://github.com/pizzamig/pot/issues/195 Crashinfo output: ``` Fatal trap 12: page fault while in kernel mode cpuid =3D 3; apic id =3D 03 fault virtual address =3D 0x18 fault code =3D supervisor read data, page not present instruction pointer =3D 0x20:0xffffffff80bffeca stack pointer =3D 0x28:0xfffffe01e0bd5820 frame pointer =3D 0x28:0xfffffe01e0bd5830 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 91596 (zfs) trap number =3D 12 panic: page fault cpuid =3D 3 time =3D 1641116990 KDB: stack backtrace: #0 0xffffffff80c40295 at kdb_backtrace+0x65 #1 0xffffffff80bf5d91 at vpanic+0x181 #2 0xffffffff80bf5b63 at panic+0x43 #3 0xffffffff810878f7 at trap_fatal+0x387 #4 0xffffffff81087966 at trap_pfault+0x66 #5 0xffffffff81086f8b at trap+0x2ab #6 0xffffffff8105b808 at calltrap+0x8 #7 0xffffffff822cabb0 at zfs_onexit_destroy+0x20 #8 0xffffffff82146768 at zfsdev_close+0x58 #9 0xffffffff80a98347 at devfs_destroy_cdevpriv+0x97 #10 0xffffffff80a9bf64 at devfs_close_f+0x64 #11 0xffffffff80b98d2b at _fdrop+0x1b #12 0xffffffff80b9c5e9 at closef+0x1d9 #13 0xffffffff80ba0697 at closefp_impl+0x77 #15 0xffffffff8105c12e at fast_syscall_common+0xf8 Uptime: 3d16h29m24s Dumping 7555 out of 65271 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..= 91% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(stru= ct pcpu, (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdown= .c:399 #2 0xffffffff80bf59bb in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:486 #3 0xffffffff80bf5e00 in vpanic (fmt=3D, ap=3D) at /usr/src/sys/kern/kern_shutdown.c:919 #4 0xffffffff80bf5b63 in panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:843 #5 0xffffffff810878f7 in trap_fatal (frame=3D0xfffffe01e0bd5760, eva=3D24) at /usr/src/sys/amd64/amd64/trap.c:915 #6 0xffffffff81087966 in trap_pfault (frame=3Dframe@entry=3D0xfffffe01e0bd= 5760,=20 usermode=3Dfalse, signo=3D, signo@entry=3D0x0,=20 ucode=3D, ucode@entry=3D0x0) at /usr/src/sys/amd64/amd64/trap.c:732 #7 0xffffffff81086f8b in trap (frame=3D0xfffffe01e0bd5760) at /usr/src/sys/amd64/amd64/trap.c:398 #8 #9 _sx_xlock (sx=3D0x0, opts=3Dopts@entry=3D0,=20 file=3D0xffffffff8239be7a "/usr/src/sys/contrib/openzfs/module/zfs/zfs_onexit.c", line=3Dline@entry= =3D89) at /usr/src/sys/kern/kern_sx.c:325 #10 0xffffffff822cabb0 in zfs_onexit_destroy (zo=3D0x0) at /usr/src/sys/contrib/openzfs/module/zfs/zfs_onexit.c:89 #11 0xffffffff82146768 in zfsdev_close (data=3D0xfffff8000822c700) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/kmod_core.c:197 #12 0xffffffff80a98347 in devfs_destroy_cdevpriv (p=3D0xfffff8051eff9b40) at /usr/src/sys/fs/devfs/devfs_vnops.c:197 #13 0xffffffff80a9bf64 in devfs_fpdrop (fp=3D0xfffff807882306e0) at /usr/src/sys/fs/devfs/devfs_vnops.c:211 #14 devfs_close_f (fp=3D0xfffff807882306e0, td=3D) at /usr/src/sys/fs/devfs/devfs_vnops.c:787 #15 0xffffffff80b98d2b in fo_close (fp=3D0xfffff807882306e0,=20 td=3D0xfffffe01e6a02300) at /usr/src/sys/sys/file.h:377 #16 _fdrop (fp=3Dfp@entry=3D0xfffff807882306e0, td=3Dtd@entry=3D0xfffffe01e= 6a02300) at /usr/src/sys/kern/kern_descrip.c:3510 #17 0xffffffff80b9c5e9 in closef (fp=3Dfp@entry=3D0xfffff807882306e0,=20 td=3Dtd@entry=3D0xfffffe01e6a02300) at /usr/src/sys/kern/kern_descrip.c= :2828 #18 0xffffffff80ba0697 in closefp_impl (fdp=3D0xfffffe01ef4134f0, fd=3D5,=20 fp=3D0xfffff807882306e0, td=3D0xfffffe01e6a02300, audit=3Dtrue) at /usr/src/sys/kern/kern_descrip.c:1271 #19 0xffffffff8108827e in syscallenter (td=3D) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:189 #20 amd64_syscall (td=3D0xfffffe01e6a02300, traced=3D0) at /usr/src/sys/amd64/amd64/trap.c:1156 #21 #22 0x00000008007bb40a in ?? () Backtrace stopped: Cannot access memory at address 0x7fffffffe9c8 (kgdb)=20 ``` --=20 You are receiving this mail because: You are the assignee for the bug.=