From nobody Sat Aug 16 01:30:09 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4c3hHp0R2dz64c2Q for ; Sat, 16 Aug 2025 01:30:30 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4c3hHn0cpyz3PPp for ; Sat, 16 Aug 2025 01:30:29 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=QZpvs04E; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2a00:1450:4864:20::52f as permitted sender) smtp.mailfrom=rick.macklem@gmail.com Received: by mail-ed1-x52f.google.com with SMTP id 4fb4d7f45d1cf-6188b72b7caso2937169a12.2 for ; Fri, 15 Aug 2025 18:30:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1755307822; x=1755912622; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=IEJZixaM38CXIw9X5QhZl/kh4QDg2WVHsAVr29Vzv0I=; b=QZpvs04ExAt/2+KBgXBy8kZHmW57Z7P6sNuNUCPnJX5SROwkBojvYvWay6/j/KYmTG qPXv2gzqXh4iaSWa2SSNUxKHF5ADql1JMELfo3w4YSa0qbptT+Qkx9IGtk5FDk1Svekl 55Gew6Pn9ImauCP/GPfRsfx7yUDLWrPlhTaGTZJjyD85mq6jDti0Hbe4TkYow9sLazzV pBY3ugQW9XPGum0JyHpZr9AhuI24H6KDiH6hMzlUwlHJDmbMhhD7fjxH0BRZMG+rqQKj E/gXA/VMQztrEOiplx4RC1TySNIgWf1YCbJHmV5GIqpHn4x8lL2YAokUnoUcpWnrG1c8 EVSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755307822; x=1755912622; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IEJZixaM38CXIw9X5QhZl/kh4QDg2WVHsAVr29Vzv0I=; b=QL3OHzzYLxCL2D5s4spajAVvWJddkuTpHiXP/nU7/yiFqye57Qusv8srZwTVmP5Iuf zvKDbUcNtAkEHu9Ka1uuJaU3ZtLeDpdxF6rPal1FJ8e+PHHk7ByconoYKvY2h/onYr9e umVJDrZCkvlske5gN2YVhdtcuNkohwgL4JVj3m57alTt+Bl2d8Omd/BXyyRXJTrFAg2W piJUKhNuunNvZ9BVWe5l6SrHyraV3ZMmrgKsU7ViKAu64EOZXWlxBqZi11REYwCPq9Z+ LdoVLnf0Qf/eC0f+jV+Uv85lr1eqQ0pvh61S6WBp/iNbrq7RU3HlwnainWVc4rR6CCMK 4G7g== X-Forwarded-Encrypted: i=1; AJvYcCX1yTx3Q4vHXnF/ycLqhGNHAVjTO+WIm2ZlvajLb3sag20fIDB5lHjZty9ylRvtOfNtidAL+dQZDp2YfHfV+ys=@freebsd.org X-Gm-Message-State: AOJu0Yw92dKv9K2eGq4x0a8JzoLc2IryWHLNZIe+vHnUUj6K447pZgYL TdnbW3Zk2ONAIWgFy0iGSvHpY4taLqy3mrmYZZZoUzdIRN/2BmdIRe/MRelZwzwCLYNX0TfY4zD KyjqqoKzEhS1l2uLIp+DPm8uUQDgmlA== X-Gm-Gg: ASbGnct9BO/ugwU/wwQw6Cd+dj7boz6fj9sRAfiqCqWzFrovKjWW/rg8Kjzp1Vl2HDu xaE81wtKoAJURDZ5Ij96A5liVN4EWMN8cz3y/qFeU/WC+sPUW5fi/PcJWXi2OvLQv1ZeEJ1V7zQ SWNal5ARi9jBhvqS2v6Exj8exVuULU/g3NPFH0RIUhP7hw27QFbhKqqlvUv6metwXDiIfICju5Z KEVFOg5RLW8Ws53kueHgHPJmsHjIG5GECkdiag= X-Google-Smtp-Source: AGHT+IHb2vyve6PkB/YKxINH4kqMMDXen1Pbcgcc0cO7zk3/29oK8yuRQhiHxRdZjLR0LqFooJh4hWqZv96aadeBuXg= X-Received: by 2002:a05:6402:5188:b0:618:a3ca:cfd2 with SMTP id 4fb4d7f45d1cf-618b051ea43mr2937454a12.15.1755307821481; Fri, 15 Aug 2025 18:30:21 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 References: <14C5523D-1F66-434F-A4D9-E14DA4BBF1E9@iitbombay.org> In-Reply-To: From: Rick Macklem Date: Fri, 15 Aug 2025 18:30:09 -0700 X-Gm-Features: Ac12FXwTZeyGP28jjPSC4pBa0s1gKf7uKFtyZ18G8tXPKf-B27_qQ5ig-NETckM Message-ID: Subject: Re: zfs related panic To: Konstantin Belousov Cc: Bakul Shah , FreeBSD Current Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TAGGED_FROM(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::52f:from]; RCVD_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4c3hHn0cpyz3PPp On Fri, Aug 15, 2025 at 6:14=E2=80=AFPM Rick Macklem wrote: > > On Fri, Aug 15, 2025 at 5:58=E2=80=AFPM Konstantin Belousov wrote: > > > > On Fri, Aug 15, 2025 at 05:41:26PM -0700, Rick Macklem wrote: > > > On Fri, Aug 15, 2025 at 5:35=E2=80=AFPM Konstantin Belousov wrote: > > > > > > > > On Fri, Aug 15, 2025 at 05:26:21PM -0700, Rick Macklem wrote: > > > > > On Fri, Aug 15, 2025 at 5:07=E2=80=AFPM Konstantin Belousov wrote: > > > > > > > > > > > > On Fri, Aug 15, 2025 at 04:51:00PM -0700, Bakul Shah wrote: > > > > > > > On Aug 15, 2025, at 3:51=E2=80=AFPM, Konstantin Belousov wrote: > > > > > > > > > > > > > > > > On Fri, Aug 15, 2025 at 11:19:55AM -0700, Bakul Shah wrote: > > > > > > > >> Is this a known bug or may be something specific on my mac= hine? > > > > > > > >> If the latter, any way to "fsck" it? FYI, the zpool is a m= irror > > > > > > > >> (two files on the host via nvme). built from c992ac621327 = commit hash > > > > > > > >> (which has other issues but they seem to be separate from = this). > > > > > > > >> I saw the same panic when I booted from a day old snapshot= . > > > > > > > >> > > > > > > > >> Note that "ls /.zfs" panics but "ls /.zfs/snapshot" doesn'= t! > > > > > > > >> > > > > > > > >> This is on a -current VM: > > > > > > > >> > > > > > > > >> root@:/ # ls .zfs > > > > > > > >> VNASSERT failed: oresid =3D=3D 0 || nresid !=3D oresid || = *(a)->a_eofflag =3D=3D 1 not true at vnode_if.c:1824 (VOP_READDIR_APV) > > > > > > > > > > > > > > > > Try this, untested. > > > > > > > > > > > > > > Thanks for the quick patch! But I am afraid it didn't help. L= et me know if you > > > > > > > want me to check things via gdb. [I have filed > > > > > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D288= 889 > > > > > > > so we can continue debugging there] > > > > > > > > > > > > > > On the console (single user, RO root): > > > > > > > # ls /.zfs > > > > > > > VNASSERT failed: oresid =3D=3D 0 || nresid !=3D oresid || *(a= )->a_eofflag =3D=3D 1 not true at vnode_if.c:1824 (VOP_READDIR_APV) > > > > > > > 0xfffff800059546e0: type VDIR state VSTATE_CONSTRUCTED op 0xf= fffffff8272cfd0 > > > > > > > usecount 1, writecount 0, refcount 1 seqc users 0 mounted= here 0 > > > > > > > hold count flags () > > > > > > > flags () > > > > > > > lock type zfs: SHARED (count 1) > > > > > > > name =3D .zfs > > > > > > > parent_id =3D 0 > > > > > > > id =3D 1 > > > > > > > panic: VOP_READDIR: eofflag not set > > > > > > > cpuid =3D 0 > > > > > > > time =3D 1755276357 > > > > > > > KDB: stack backtrace: > > > > > > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0= xfffffe0053f83af0 > > > > > > > vpanic() at vpanic+0x136/frame 0xfffffe0053f83c20 > > > > > > > panic() at panic+0x43/frame 0xfffffe0053f83c80 > > > > > > > VOP_READDIR_APV() at VOP_READDIR_APV+0x205/frame 0xfffffe0053= f83cd0 > > > > > > > kern_getdirentries() at kern_getdirentries+0x228/frame 0xffff= fe0053f83dd0 > > > > > > > sys_getdirentries() at sys_getdirentries+0x29/frame 0xfffffe0= 053f83e00 > > > > > > > amd64_syscall() at amd64_syscall+0x169/frame 0xfffffe0053f83f= 30 > > > > > > > fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfff= ffe0053f83f30 > > > > > > > --- syscall (554, FreeBSD ELF64, getdirentries), rip =3D 0x33= 1339f976aa, rsp =3D 0x33133631ade8, rbp =3D 0x33133631ae20 --- > > > > > > > KDB: enter: panic > > > > > > > [ thread pid 23 tid 100211 ] > > > > > > > Stopped at kdb_enter+0x33: movq $0,0x12313e2(%rip) > > > > > > > db> > > > > > > > > > > > > > > Running gdb on the host (attached to tcp port): > > > > > > > #16 0xffffffff80b7992b in vpanic ( > > > > > > > fmt=3D0xffffffff812ddf30 "VOP_READDIR: eofflag not set", > > > > > > > ap=3Dap@entry=3D0xfffffe0053f83c60) > > > > > > > at /home/FreeBSD/current/sys/kern/kern_shutdown.c:962 > > > > > > > #17 0xffffffff80b79793 in panic ( > > > > > > > fmt=3D0xffffffff81d9eab0 "\304\372\032\201\3= 77\377\377\377") > > > > > > > at /home/FreeBSD/current/sys/kern/kern_shutdown.c:887 > > > > > > > #18 0xffffffff81195fd5 in VOP_READDIR_APV (vop=3D, > > > > > > > a=3Da@entry=3D0xfffffe0053f83d30) at vnode_if.c:1824 > > > > > > > #19 0xffffffff80c95e58 in VOP_READDIR (vp=3D0xfffff800059546e= 0, > > > > > > > uio=3D0xfffffe0053f83d00, cred=3D, eofflag= =3D0xfffffe0053f83d6c, > > > > > > > ncookies=3D0x0, cookies=3D0x0) at ./vnode_if.h:972 > > > > > > From this frame, do > > > > > > p *vp > > > > > > and > > > > > > p *(vp->v_op) > > > > > > I am mostly interested what is the .vop_readdir fp points to. > > > > > I think the problem is that, for this case, ZFS replies with eoff= lag > > > > > =3D=3D -1 instead > > > > > of 1. (I don't know if you want to change the ASSERT or try to fi= x ZFS > > > > > to not do this?) > > > > > > > > Where do you see it? I mean the '-1' set to *eofp. > > > I saw it in a printf() after VOP_READDIR(). However, a subsequent > > > test showed 0. > > > --> The first time I was printing out for non-ZFS, so there was a fai= r amount > > > of other printf()s being logged. Then I limited it to ZFS. > > > > > > Anyhow, ZFS seems to get eof wrong when it is already at eof. > > > > > > I'll take a look at the ZFS code, rick > > > > > > > Below is what I gathered so far. > > > > diff --git a/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c b/s= ys/contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c > > index 61d0bb26d1e5..c191b2abf6cc 100644 > > --- a/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c > > +++ b/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c > > @@ -674,6 +674,7 @@ zfsctl_root_readdir(struct vop_readdir_args *ap) > > zfs_uio_t uio; > > int *eofp =3D ap->a_eofflag; > > off_t dots_offset; > > + ssize_t orig_resid; > > int error; > > > > zfs_uio_init(&uio, ap->a_uio); > > @@ -688,14 +689,20 @@ zfsctl_root_readdir(struct vop_readdir_args *ap) > > * count to return is 0. > > */ > > if (zfs_uio_offset(&uio) =3D=3D 3 * sizeof (entry)) { > > + if (eofp !=3D NULL) > > + *eofp =3D 1; > Yea, I was just going to try the same thing. I think it probably fixes > the assert. This fixes the simple test case I am doing. > (Of course, it might take a while to get this in ZFS, so you might have t= o drop > the assert until it downstreams from OpenZFS?) > > > return (0); > > } > > > > + orig_resid =3D zfs_uio_resid(&uio); > > error =3D sfs_readdir_common(zfsvfs->z_root, ZFSCTL_INO_ROOT, a= p, &uio, > > &dots_offset); > > if (error !=3D 0) { > > - if (error =3D=3D ENAMETOOLONG) /* ran out of destinatio= n space */ > > + if (error =3D=3D ENAMETOOLONG) { /* ran out of destinat= ion space */ > > error =3D 0; > > + if (orig_resid =3D=3D zfs_uio_resid(&uio) && eo= fp !=3D NULL) > > + *eofp =3D 1; > If it hasn't filled in an entry, shouldn't it be an error return and not > setting "error =3D 0;"? > > > + } > > return (error); > > } > > if (zfs_uio_offset(&uio) !=3D dots_offset) > > @@ -710,8 +717,11 @@ zfsctl_root_readdir(struct vop_readdir_args *ap) > > entry.d_reclen =3D sizeof (entry); > > error =3D vfs_read_dirent(ap, &entry, zfs_uio_offset(&uio)); > > if (error !=3D 0) { > > - if (error =3D=3D ENAMETOOLONG) > > + if (error =3D=3D ENAMETOOLONG) { > > error =3D 0; > > + if (orig_resid =3D=3D zfs_uio_resid(&uio) && eo= fp !=3D NULL) > > + *eofp =3D 1; > Ditto above. > > + } > > return (SET_ERROR(error)); > > } > > if (eofp !=3D NULL) > > @@ -1056,17 +1066,22 @@ zfsctl_snapdir_readdir(struct vop_readdir_args = *ap) > > zfs_uio_t uio; > > int *eofp =3D ap->a_eofflag; > > off_t dots_offset; > > + ssize_t orig_resid; > > int error; > > > > zfs_uio_init(&uio, ap->a_uio); > > + orig_resid =3D zfs_uio_resid(&uio); > > > > ASSERT3S(vp->v_type, =3D=3D, VDIR); > > > > error =3D sfs_readdir_common(ZFSCTL_INO_ROOT, ZFSCTL_INO_SNAPDI= R, ap, > > &uio, &dots_offset); > > if (error !=3D 0) { > > - if (error =3D=3D ENAMETOOLONG) /* ran out of destinatio= n space */ > > + if (error =3D=3D ENAMETOOLONG) { /* ran out of destinat= ion space */ > > error =3D 0; > > + if (orig_resid =3D=3D zfs_uio_resid(&uio) && eo= fp !=3D NULL) > > + *eofp =3D 1; > > + } > Ditto again. > > return (error); > > } > > > > @@ -1084,7 +1099,8 @@ zfsctl_snapdir_readdir(struct vop_readdir_args *a= p) > > dsl_pool_config_exit(dmu_objset_pool(zfsvfs->z_os), FTA= G); > > if (error !=3D 0) { > > if (error =3D=3D ENOENT) { > > - if (eofp !=3D NULL) > > + if (orig_resid =3D=3D zfs_uio_resid(&ui= o) && > > + eofp !=3D NULL) > > *eofp =3D 1; > > error =3D 0; > > } > > @@ -1099,8 +1115,12 @@ zfsctl_snapdir_readdir(struct vop_readdir_args *= ap) > > entry.d_reclen =3D sizeof (entry); > > error =3D vfs_read_dirent(ap, &entry, zfs_uio_offset(&u= io)); > > if (error !=3D 0) { > > - if (error =3D=3D ENAMETOOLONG) > > + if (error =3D=3D ENAMETOOLONG) { > > error =3D 0; > > + if (orig_resid =3D=3D zfs_uio_resid(&ui= o) && > > + eofp !=3D NULL) > > + *eofp =3D 1; > Ditto again. > > + } > > zfs_exit(zfsvfs, FTAG); > > return (SET_ERROR(error)); > > } > > diff --git a/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c b= /sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c > > index 1813c411b013..79e808a3ab3d 100644 > > --- a/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c > > +++ b/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c > > @@ -1695,6 +1695,7 @@ zfs_readdir(vnode_t *vp, zfs_uio_t *uio, cred_t *= cr, int *eofp, > > objset_t *os; > > caddr_t outbuf; > > size_t bufsize; > > + ssize_t orig_resid; > > zap_cursor_t zc; > > zap_attribute_t *zap; > > uint_t bytes_wanted; > > @@ -1735,7 +1736,7 @@ zfs_readdir(vnode_t *vp, zfs_uio_t *uio, cred_t *= cr, int *eofp, > > /* > > * Quit if directory has been removed (posix) > > */ > > - if ((*eofp =3D zp->z_unlinked) !=3D 0) { > > + if ((*eofp =3D (zp->z_unlinked !=3D 0)) !=3D 0) { > > zfs_exit(zfsvfs, FTAG); > > return (0); > > } > > @@ -1743,6 +1744,7 @@ zfs_readdir(vnode_t *vp, zfs_uio_t *uio, cred_t *= cr, int *eofp, > > error =3D 0; > > os =3D zfsvfs->z_os; > > offset =3D zfs_uio_offset(uio); > > + orig_resid =3D zfs_uio_resid(uio); > > prefetch =3D zp->z_zn_prefetch; > > zap =3D zap_attribute_long_alloc(); > > > > @@ -1933,6 +1935,8 @@ zfs_readdir(vnode_t *vp, zfs_uio_t *uio, cred_t *= cr, int *eofp, > > *cookies =3D NULL; > > *ncookies =3D 0; > > } > > + if (error =3D=3D 0 && zfs_uio_resid(uio) !=3D orig_resid) > > + *eofp =3D 1; > I looked at this earlier and I don't this is needed. When it breaks out > of the loop, it has set eof if it got ENOENT from zap_cursor_retrieve(), > which I think is the only case that will return 0 and not an error? > (But it looks harmless to me.) > > > return (error); > > } > > > >