From nobody Mon Sep 11 23:17:52 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Rl2y32jWrz4t5Fg for ; Mon, 11 Sep 2023 23:30:19 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from omta001.cacentral1.a.cloudfilter.net (omta001.cacentral1.a.cloudfilter.net [3.97.99.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Rl2y23m72z4Wnd; Mon, 11 Sep 2023 23:30:18 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of cy.schubert@cschubert.com has no SPF policy when checking 3.97.99.32) smtp.mailfrom=cy.schubert@cschubert.com; dmarc=none Received: from shw-obgw-4002a.ext.cloudfilter.net ([10.228.9.250]) by cmsmtp with ESMTP id ffa6qWDETLAoIfqAEqoKzC; Mon, 11 Sep 2023 23:17:54 +0000 Received: from spqr.komquats.com ([70.66.152.170]) by cmsmtp with ESMTPA id fqADqtmMMyAOefqAEqrWq2; Mon, 11 Sep 2023 23:17:54 +0000 X-Authority-Analysis: v=2.4 cv=e5oV9Il/ c=1 sm=1 tr=0 ts=64ffa022 a=y8EK/9tc/U6QY+pUhnbtgQ==:117 a=y8EK/9tc/U6QY+pUhnbtgQ==:17 a=kj9zAlcOel0A:10 a=zNV7Rl7Rt7sA:10 a=YxBL1-UpAAAA:8 a=6I5d2MoRAAAA:8 a=EkcXrb_YAAAA:8 a=mHWwJs-SP0RyW_hiVBcA:9 a=CjuIK1q_8ugA:10 a=Ia-lj3WSrqcvXOmTRaiG:22 a=IjZwj45LgO3ly-622nXo:22 a=LK5xJRSDVpKd5WXXoEvA:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTP id DE6492A0; Mon, 11 Sep 2023 16:17:52 -0700 (PDT) Received: by slippy.cwsent.com (Postfix, from userid 1000) id B2B8C1AC; Mon, 11 Sep 2023 16:17:52 -0700 (PDT) X-Mailer: exmh version 2.9.0 11/07/2018 with nmh-1.8+dev Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: amotin@freebsd.org, mm@freebsd.org, current@freebsd.org Subject: ZFS Panics Still List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 11 Sep 2023 16:17:52 -0700 Message-Id: <20230911231752.B2B8C1AC@slippy.cwsent.com> X-CMAE-Envelope: MS4xfGNb79UbQVSLsMzNKgAAXye9yoBRuJiwKeFlNzNnQQV/H0IWHhzB+8AMPHIFhn/rcFLi4gpTem+wWIfuxfyudgbsf7ZnSRev5iDce0gp15FFFq3+39my 8tAwSR3d+lx2C5DCoCZx9pJip8WOO+at63dY8j3+JF8TglxBQJjuhwmMpRynXRrDtgMHC+00pfcNdXeaTqF6k1xKfN39lF2JqKkH5Pk97UuvahewNb1nGuOh R3TqnWjENE0jIIBzZ+kf5A== X-Spamd-Bar: + X-Spamd-Result: default: False [1.80 / 15.00]; R_BAD_CTE_7BIT(3.50)[7bit]; AUTH_NA(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; MV_CASE(0.50)[]; RCVD_IN_DNSWL_LOW(-0.10)[3.97.99.32:from]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[current@freebsd.org]; DMARC_NA(0.00)[cschubert.com]; FROM_HAS_DN(0.00)[]; REPLYTO_EQ_FROM(0.00)[]; HAS_REPLYTO(0.00)[Cy.Schubert@cschubert.com]; R_SPF_NA(0.00)[no SPF record]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_NONE(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:16509, ipnet:3.96.0.0/15, country:US]; GREYLIST(0.00)[pass,body] X-Rspamd-Queue-Id: 4Rl2y23m72z4Wnd Hi, One of my machines, running poudriere building i386 packges, panics, below, at termination of poudriere while poudriere is cleaning up performing zfs umount. I just happenedto catch the machine to see poudriere just having completed and umounting filesystems. panic: vm_page_dequeue_deferred: page 0xfffffe000b7e9748 has unexpected queue state cpuid = 1 time = 1694472686 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00bebac7d0 vpanic() at vpanic+0x132/frame 0xfffffe00bebac900 panic() at panic+0x43/frame 0xfffffe00bebac960 vm_page_dequeue_deferred() at vm_page_dequeue_deferred+0xb2/frame 0xfffffe00bebac970 vm_page_free_prep() at vm_page_free_prep+0x11b/frame 0xfffffe00bebac990 vm_page_free_toq() at vm_page_free_toq+0x12/frame 0xfffffe00bebac9c0 vm_object_page_remove() at vm_object_page_remove+0xb6/frame 0xfffffe00bebaca20 bufobj_invalbuf() at bufobj_invalbuf+0x198/frame 0xfffffe00bebaca80 vgonel() at vgonel+0x2ce/frame 0xfffffe00bebacaf0 vflush() at vflush+0x3ad/frame 0xfffffe00bebacc40 zfs_umount() at zfs_umount+0xca/frame 0xfffffe00bebacc80 dounmount() at dounmount+0x7b5/frame 0xfffffe00bebaccf0 kern_unmount() at kern_unmount+0x2eb/frame 0xfffffe00bebace00 amd64_syscall() at amd64_syscall+0x138/frame 0xfffffe00bebacf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00bebacf30 --- syscall (22, FreeBSD ELF64, unmount), rip = 0x36bbc3e23dba, rsp = 0x36bbbb4f08e8, rbp = 0x36bbbb4f08f0 --- Uptime: 46m16s Dumping 3452 out of 7996 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% __curthread () at /opt/src/git-src/sys/amd64/include/pcpu_aux.h:57 57 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu, (kgdb) bt #0 __curthread () at /opt/src/git-src/sys/amd64/include/pcpu_aux.h:57 #1 doadump (textdump=textdump@entry=1) at /opt/src/git-src/sys/kern/kern_shutdown.c:405 #2 0xffffffff806c20b0 in kern_reboot (howto=260) at /opt/src/git-src/sys/kern/kern_shutdown.c:526 #3 0xffffffff806c25af in vpanic ( fmt=0xffffffff80b5e2e6 "%s: page %p has unexpected queue state", ap=ap@entry=0xfffffe00bebac940) at /opt/src/git-src/sys/kern/kern_shutdown.c:970 #4 0xffffffff806c2353 in panic (fmt=) at /opt/src/git-src/sys/kern/kern_shutdown.c:894 #5 0xffffffff809db0c2 in vm_page_dequeue_deferred (m=, m@entry=0xfffffe000b7e9748) at /opt/src/git-src/sys/vm/vm_page.c:3790 #6 0xffffffff809de5fb in vm_page_free_prep (m=m@entry=0xfffffe000b7e9748) at /opt/src/git-src/sys/vm/vm_page.c:3928 #7 0xffffffff809d6162 in vm_page_free_toq (m=, m@entry=0xfffffe000b7e9748) at /opt/src/git-src/sys/vm/vm_page.c:3970 #8 0xffffffff809d614b in vm_page_free (m=, m@entry=0xfffffe000b7e9748) at /opt/src/git-src/sys/vm/vm_page.c:1328 #9 0xffffffff809d0f16 in vm_object_page_remove (object=0xfffff8013edc0108, start=0, end=0, options=1) at /opt/src/git-src/sys/vm/vm_object.c:2157 #10 0xffffffff807b6a28 in bufobj_invalbuf (bo=0xfffff800665c50e0, flags=1, slpflag=slpflag@entry=0, slptimeo=slptimeo@entry=0) --Type for more, q to quit, c to continue without paging--c at /opt/src/git-src/sys/kern/vfs_subr.c:2156 #11 0xffffffff807ba2ee in vgonel (vp=vp@entry=0xfffff800665c5000) at /opt/src/git-src/sys/kern/vfs_subr.c:2187 #12 0xffffffff807b999d in vflush (mp=mp@entry=0xfffffe00da21cb00, rootrefs=rootrefs@entry=0, flags=flags@entry=2, td=td@entry=0xfffffe00c4610720) at /opt/src/git-src/sys/kern/vfs_subr.c:3939 #13 0xffffffff83871bea in zfs_umount (vfsp=0xfffffe00da21cb00, vfsp@entry=, fflag=, fflag@entry=) at /opt/src/git-src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vfsops.c:1708 #14 0xffffffff807ae405 in dounmount (mp=, mp@entry=0xfffffe00da21cb00, flags=flags@entry=524288, td=td@entry=0xfffffe00c4610720) at /opt/src/git-src/sys/kern/vfs_mount.c:2327 #15 0xffffffff807adbeb in kern_unmount (td=0xfffffe00c4610720, path=0x5a7647a1a3c0 , flags=524288) at /opt/src/git-src/sys/kern/vfs_mount.c:1785 #16 0xffffffff80a741b8 in syscallenter (td=) at /opt/src/git-src/sys/amd64/amd64/../../kern/subr_syscall.c:187 #17 amd64_syscall (td=0xfffffe00c4610720, traced=0) at /opt/src/git-src/sys/amd64/amd64/trap.c:1197 #18 #19 0x000036bbc3e23dba in ?? () Backtrace stopped: Cannot access memory at address 0x36bbbb4f08e8 At frame 6 *m contains: (kgdb) p *m $5 = {plinks = {q = {tqe_next = 0xffffffffffffffff, tqe_prev = 0xffffffffffffffff}, s = {ss = { sle_next = 0xffffffffffffffff}}, memguard = {p = 18446744073709551615, v = 18446744073709551615}, uma = {slab = 0xffffffffffffffff, zone = 0xffffffffffffffff}}, listq = {tqe_next = 0xffffffffffffffff, tqe_prev = 0xffffffffffffffff}, object = 0x0, pindex = 14685, phys_addr = 7595216896, md = {pv_list = {tqh_first = 0x0, tqh_last = 0xfffffe000b7e9780}, pv_gen = 20, pat_mode = 6}, ref_count = 0, busy_lock = 4294967294, a = {{flags = 16, queue = 255 '\377', act_count = 0 '\000'}, _bits = 16711696}, order = 13 '\r', pool = 0 '\000', flags = 0 '\000', oflags = 0 '\000', psind = 0 '\000', segind = 5 '\005', valid = 255 '\377', dirty = 0 '\000'} (kgdb) At frame 13 *vfsp contains: $7 = {mnt_vfs_ops = 1, mnt_kern_flag = 1090847177, mnt_flag = 268439568, mnt_pcpu = 0xfffffe010d84cbc0, mnt_rootvnode = 0x0, mnt_vnodecovered = 0xfffff8008dc691c0, mnt_op = 0xffffffff83bb3080 , mnt_vfc = 0xffffffff83bb3228 , mnt_mtx = {lock_object = { lo_name = 0xffffffff80abf68d "struct mount mtx", lo_flags = 16973824, lo_data = 0, lo_witness = 0xfffff8021fd75b00}, mtx_lock = 0}, mnt_gen = 1, mnt_list = {tqe_next = 0x0, tqe_prev = 0xfffffe00c45f2168}, mnt_syncer = 0x0, mnt_ref = 27, mnt_nvnodelist = { tqh_first = 0xfffff800665c5000, tqh_last = 0xfffff8007cc90aa8}, mnt_nvnodelistsize = 24, mnt_writeopcount = 1, mnt_opt = 0xfffff80084d56cd0, mnt_optnew = 0x0, mnt_stat = {f_version = 538182936, f_type = 222, f_flags = 268439568, f_bsize = 512, f_iosize = 131072, f_blocks = 251997486, f_bfree = 248369646, f_bavail = 248369646, f_files = 248516350, f_ffree = 248369646, f_syncwrites = 0, f_asyncwrites = 0, f_syncreads = 0, f_asyncreads = 0, f_nvnodelistsize = 113, f_spare0 = 0, f_spare = {0, 0, 0, 0, 0, 0, 0, 0, 0}, f_namemax = 255, f_owner = 0, f_fsid = {val = {-313067424, 1444670686}}, f_charspare = '\000' , f_fstypename = "zfs", '\000' , f_mntfromname = "bob/poudriere/bob/jails/HEADi386-new-ports-ref/04", '\000' , f_mntonname = "/poudriere/bob/data/.m/HEADi386-new-ports/04", '\000' }, mnt_cred = 0xfffff800c83cb200, mnt_data = 0xfffff800b713e000, mnt_time = 0, mnt_iosize_max = 65536, mnt_export = 0x0, mnt_label = 0x0, mnt_hashseed = 1242221059, mnt_lockref = 0, mnt_secondary_writes = 0, mnt_secondary_accwrites = 0, mnt_susp_owner = 0x0, mnt_exjail = 0x0, mnt_gjprovider = 0x0, mnt_listmtx = {lock_object = { lo_name = 0xffffffff80b1539e "struct mount vlist mtx", lo_flags = 16973824, lo_data = 0, lo_witness = 0xfffff8021fd82a80}, mtx_lock = 0}, mnt_lazyvnodelist = {tqh_first = 0x0, tqh_last = 0xfffffe00da21d550}, mnt_lazyvnodelistsize = 0, mnt_upper_pending = 0, mnt_explock = {lock_object = { lo_name = 0xffffffff80b6167f "explock", lo_flags = 108199936, lo_data = 0, lo_witness = 0xfffff8021fd82880}, lk_lock = 1, lk_exslpfail AÆí$= 0, lk_pri = 64, lk_timo = 0}, mnt_uppers = { tqh_first = 0x0, tqh_last = 0xfffffe00da21d590}, mnt_notify = { tqh_first = 0x0, tqh_last = 0xfffffe00da21d5a0}, mnt_taskqueue_link = { stqe_next = 0x0}, mnt_taskqueue_flags = 0, mnt_unmount_retries = 0} -- Cheers, Cy Schubert FreeBSD UNIX: Web: https://FreeBSD.org NTP: Web: https://nwtime.org e^(i*pi)+1=0