From nobody Sat Aug 16 01:14:39 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4c3gxt4XbTz64b02 for ; Sat, 16 Aug 2025 01:14:58 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4c3gxt0ZXtz3MQT for ; Sat, 16 Aug 2025 01:14:58 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x52a.google.com with SMTP id 4fb4d7f45d1cf-6188b5b11b2so3141710a12.0 for ; Fri, 15 Aug 2025 18:14:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1755306891; x=1755911691; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=1NEiPpDIWfTK69wog8TycqDIPUAxWqOrq48j7TMgXSs=; b=UGvbnMnfH6k9gg+4NKPdEUM2143a39mVXKzCqhZcDKLAkqhuaoe+6lWMe41/9OV1ip 4cSlbXqrwnC3NAyWRSdxApQo7wMTlrTVQZ3QcIItJDSfEjna0nqTz/qQ/BI09jMgo5JG tF+yibou8Y2oYc472xyoiyWdcRW07mCRpP2IMg2f9X0V/scd0zFGCj9woGlQXR8ms1rf RbV8JVdVjiVGkelz+qBMuvFPfVw6jAwevrFvXgQWVZuReYWSZNSvXFjdYh4GYGczwuXN FrFx670dUQ/NRrK/jx7/5n5grx1QNc1J7ImdmBvfdmeSUv2mD/u2Y6CGK5UPOKCwpvaL atJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755306891; x=1755911691; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1NEiPpDIWfTK69wog8TycqDIPUAxWqOrq48j7TMgXSs=; b=XdNJtuJX/4HXds9V5pNkW4Brqnzznvvt/+7xHkqM8y/65y+OMobEZ7jRe0Ba0Mpm3k xWnvy+vPx36I2Z79tEBnwUjGxJhNey26kIYYaQxrHz7ND0i8mdlYjxCCim7IUE37ct+n Xg0cCuOyfJ+U8zzN6gYGWXEdlo26Swsm5a1+6BARStDHIgXvz2Ub0HsOKwGBulG/Ea78 294VyqKGzc5uiefnbPV89TGg9S78WeNQr71JlDXi371dZ/6D3T6kl2eu2aWyyTSVHweS ujDb5bXGt0qu4PaTXpk9LiDIyoYa9oRy97s9Q5szhrfbvTV+QqpStGKMVVIHw4RqhQRO kzYw== X-Forwarded-Encrypted: i=1; AJvYcCXHscJ1YQmELrmbZemSpL+bN4HFA7S1NfJQnHhykJoZFEoX+qwaJW3P+FUHQjWcR6yHTWdMVrxL8aCyejzY17k=@freebsd.org X-Gm-Message-State: AOJu0Yy6pddSUwaMSSQCP5LMbWlWMJiY98kZDphuT7dvB7FNU+JcyHj6 tSYKuVwfOzbfv2bdagCUZxu1sRJ4INgLDTDe4ZZMjL7gkLCyzrOOZ7RdFQVZAbocEp2I3n56XeW AtQUJkP5LbOuWE8j9k64rJUS3RyUoTscQ X-Gm-Gg: ASbGncuqX3ctorIDhDecnxFLSoqRJziuudHQ4zOO2xAp+RTACmsp/2TmEBy9yldPSnY Odd1xi6+oAGZJx9U2tWUKsy/tLskC+jWL2Qw1ASJUtes7UYYZ7wJWpasPygz6SzScPJJdOls33j d+EKBuEB7eJa7IZGCHqOjYUXdEt5MK88yHwts4DQAZedY6RG7GW/xBykMh5/8qLCevXVilH+JZK PricPd0nqaQ039Wp5WFKuOY+xTqPkbZfgR0020PLNbdfQmnbQ== X-Google-Smtp-Source: AGHT+IHp5JUZdF7c2wvJF89H97DIZAQT74akgIQVh2HwzOZAB02Cfq/MjFla0QJK5Eg7a1sj/X/gNrl7fDxfhmYeZ50= X-Received: by 2002:a05:6402:2345:b0:618:af07:9e09 with SMTP id 4fb4d7f45d1cf-619bf1d686dmr781710a12.17.1755306890989; Fri, 15 Aug 2025 18:14:50 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 References: <14C5523D-1F66-434F-A4D9-E14DA4BBF1E9@iitbombay.org> In-Reply-To: From: Rick Macklem Date: Fri, 15 Aug 2025 18:14:39 -0700 X-Gm-Features: Ac12FXz2jPLmkFmNejZEVpZFW29WoAZ4Qi1mgr0jAiG58amjr1NmgJD9qVrPFJg Message-ID: Subject: Re: zfs related panic To: Konstantin Belousov Cc: Bakul Shah , FreeBSD Current Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4c3gxt0ZXtz3MQT On Fri, Aug 15, 2025 at 5:58=E2=80=AFPM Konstantin Belousov wrote: > > On Fri, Aug 15, 2025 at 05:41:26PM -0700, Rick Macklem wrote: > > On Fri, Aug 15, 2025 at 5:35=E2=80=AFPM Konstantin Belousov wrote: > > > > > > On Fri, Aug 15, 2025 at 05:26:21PM -0700, Rick Macklem wrote: > > > > On Fri, Aug 15, 2025 at 5:07=E2=80=AFPM Konstantin Belousov wrote: > > > > > > > > > > On Fri, Aug 15, 2025 at 04:51:00PM -0700, Bakul Shah wrote: > > > > > > On Aug 15, 2025, at 3:51=E2=80=AFPM, Konstantin Belousov wrote: > > > > > > > > > > > > > > On Fri, Aug 15, 2025 at 11:19:55AM -0700, Bakul Shah wrote: > > > > > > >> Is this a known bug or may be something specific on my machi= ne? > > > > > > >> If the latter, any way to "fsck" it? FYI, the zpool is a mir= ror > > > > > > >> (two files on the host via nvme). built from c992ac621327 co= mmit hash > > > > > > >> (which has other issues but they seem to be separate from th= is). > > > > > > >> I saw the same panic when I booted from a day old snapshot. > > > > > > >> > > > > > > >> Note that "ls /.zfs" panics but "ls /.zfs/snapshot" doesn't! > > > > > > >> > > > > > > >> This is on a -current VM: > > > > > > >> > > > > > > >> root@:/ # ls .zfs > > > > > > >> VNASSERT failed: oresid =3D=3D 0 || nresid !=3D oresid || *(= a)->a_eofflag =3D=3D 1 not true at vnode_if.c:1824 (VOP_READDIR_APV) > > > > > > > > > > > > > > Try this, untested. > > > > > > > > > > > > Thanks for the quick patch! But I am afraid it didn't help. Let= me know if you > > > > > > want me to check things via gdb. [I have filed > > > > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D28888= 9 > > > > > > so we can continue debugging there] > > > > > > > > > > > > On the console (single user, RO root): > > > > > > # ls /.zfs > > > > > > VNASSERT failed: oresid =3D=3D 0 || nresid !=3D oresid || *(a)-= >a_eofflag =3D=3D 1 not true at vnode_if.c:1824 (VOP_READDIR_APV) > > > > > > 0xfffff800059546e0: type VDIR state VSTATE_CONSTRUCTED op 0xfff= fffff8272cfd0 > > > > > > usecount 1, writecount 0, refcount 1 seqc users 0 mountedhe= re 0 > > > > > > hold count flags () > > > > > > flags () > > > > > > lock type zfs: SHARED (count 1) > > > > > > name =3D .zfs > > > > > > parent_id =3D 0 > > > > > > id =3D 1 > > > > > > panic: VOP_READDIR: eofflag not set > > > > > > cpuid =3D 0 > > > > > > time =3D 1755276357 > > > > > > KDB: stack backtrace: > > > > > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xf= ffffe0053f83af0 > > > > > > vpanic() at vpanic+0x136/frame 0xfffffe0053f83c20 > > > > > > panic() at panic+0x43/frame 0xfffffe0053f83c80 > > > > > > VOP_READDIR_APV() at VOP_READDIR_APV+0x205/frame 0xfffffe0053f8= 3cd0 > > > > > > kern_getdirentries() at kern_getdirentries+0x228/frame 0xfffffe= 0053f83dd0 > > > > > > sys_getdirentries() at sys_getdirentries+0x29/frame 0xfffffe005= 3f83e00 > > > > > > amd64_syscall() at amd64_syscall+0x169/frame 0xfffffe0053f83f30 > > > > > > fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffff= e0053f83f30 > > > > > > --- syscall (554, FreeBSD ELF64, getdirentries), rip =3D 0x3313= 39f976aa, rsp =3D 0x33133631ade8, rbp =3D 0x33133631ae20 --- > > > > > > KDB: enter: panic > > > > > > [ thread pid 23 tid 100211 ] > > > > > > Stopped at kdb_enter+0x33: movq $0,0x12313e2(%rip) > > > > > > db> > > > > > > > > > > > > Running gdb on the host (attached to tcp port): > > > > > > #16 0xffffffff80b7992b in vpanic ( > > > > > > fmt=3D0xffffffff812ddf30 "VOP_READDIR: eofflag not set", > > > > > > ap=3Dap@entry=3D0xfffffe0053f83c60) > > > > > > at /home/FreeBSD/current/sys/kern/kern_shutdown.c:962 > > > > > > #17 0xffffffff80b79793 in panic ( > > > > > > fmt=3D0xffffffff81d9eab0 "\304\372\032\201\377= \377\377\377") > > > > > > at /home/FreeBSD/current/sys/kern/kern_shutdown.c:887 > > > > > > #18 0xffffffff81195fd5 in VOP_READDIR_APV (vop=3D, > > > > > > a=3Da@entry=3D0xfffffe0053f83d30) at vnode_if.c:1824 > > > > > > #19 0xffffffff80c95e58 in VOP_READDIR (vp=3D0xfffff800059546e0, > > > > > > uio=3D0xfffffe0053f83d00, cred=3D, eofflag= =3D0xfffffe0053f83d6c, > > > > > > ncookies=3D0x0, cookies=3D0x0) at ./vnode_if.h:972 > > > > > From this frame, do > > > > > p *vp > > > > > and > > > > > p *(vp->v_op) > > > > > I am mostly interested what is the .vop_readdir fp points to. > > > > I think the problem is that, for this case, ZFS replies with eoffla= g > > > > =3D=3D -1 instead > > > > of 1. (I don't know if you want to change the ASSERT or try to fix = ZFS > > > > to not do this?) > > > > > > Where do you see it? I mean the '-1' set to *eofp. > > I saw it in a printf() after VOP_READDIR(). However, a subsequent > > test showed 0. > > --> The first time I was printing out for non-ZFS, so there was a fair = amount > > of other printf()s being logged. Then I limited it to ZFS. > > > > Anyhow, ZFS seems to get eof wrong when it is already at eof. > > > > I'll take a look at the ZFS code, rick > > > > Below is what I gathered so far. > > diff --git a/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c b/sys= /contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c > index 61d0bb26d1e5..c191b2abf6cc 100644 > --- a/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c > +++ b/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c > @@ -674,6 +674,7 @@ zfsctl_root_readdir(struct vop_readdir_args *ap) > zfs_uio_t uio; > int *eofp =3D ap->a_eofflag; > off_t dots_offset; > + ssize_t orig_resid; > int error; > > zfs_uio_init(&uio, ap->a_uio); > @@ -688,14 +689,20 @@ zfsctl_root_readdir(struct vop_readdir_args *ap) > * count to return is 0. > */ > if (zfs_uio_offset(&uio) =3D=3D 3 * sizeof (entry)) { > + if (eofp !=3D NULL) > + *eofp =3D 1; Yea, I was just going to try the same thing. I think it probably fixes the assert. (Of course, it might take a while to get this in ZFS, so you might have to = drop the assert until it downstreams from OpenZFS?) > return (0); > } > > + orig_resid =3D zfs_uio_resid(&uio); > error =3D sfs_readdir_common(zfsvfs->z_root, ZFSCTL_INO_ROOT, ap,= &uio, > &dots_offset); > if (error !=3D 0) { > - if (error =3D=3D ENAMETOOLONG) /* ran out of destination = space */ > + if (error =3D=3D ENAMETOOLONG) { /* ran out of destinatio= n space */ > error =3D 0; > + if (orig_resid =3D=3D zfs_uio_resid(&uio) && eofp= !=3D NULL) > + *eofp =3D 1; If it hasn't filled in an entry, shouldn't it be an error return and not setting "error =3D 0;"? > + } > return (error); > } > if (zfs_uio_offset(&uio) !=3D dots_offset) > @@ -710,8 +717,11 @@ zfsctl_root_readdir(struct vop_readdir_args *ap) > entry.d_reclen =3D sizeof (entry); > error =3D vfs_read_dirent(ap, &entry, zfs_uio_offset(&uio)); > if (error !=3D 0) { > - if (error =3D=3D ENAMETOOLONG) > + if (error =3D=3D ENAMETOOLONG) { > error =3D 0; > + if (orig_resid =3D=3D zfs_uio_resid(&uio) && eofp= !=3D NULL) > + *eofp =3D 1; Ditto above. > + } > return (SET_ERROR(error)); > } > if (eofp !=3D NULL) > @@ -1056,17 +1066,22 @@ zfsctl_snapdir_readdir(struct vop_readdir_args *a= p) > zfs_uio_t uio; > int *eofp =3D ap->a_eofflag; > off_t dots_offset; > + ssize_t orig_resid; > int error; > > zfs_uio_init(&uio, ap->a_uio); > + orig_resid =3D zfs_uio_resid(&uio); > > ASSERT3S(vp->v_type, =3D=3D, VDIR); > > error =3D sfs_readdir_common(ZFSCTL_INO_ROOT, ZFSCTL_INO_SNAPDIR,= ap, > &uio, &dots_offset); > if (error !=3D 0) { > - if (error =3D=3D ENAMETOOLONG) /* ran out of destination = space */ > + if (error =3D=3D ENAMETOOLONG) { /* ran out of destinatio= n space */ > error =3D 0; > + if (orig_resid =3D=3D zfs_uio_resid(&uio) && eofp= !=3D NULL) > + *eofp =3D 1; > + } Ditto again. > return (error); > } > > @@ -1084,7 +1099,8 @@ zfsctl_snapdir_readdir(struct vop_readdir_args *ap) > dsl_pool_config_exit(dmu_objset_pool(zfsvfs->z_os), FTAG)= ; > if (error !=3D 0) { > if (error =3D=3D ENOENT) { > - if (eofp !=3D NULL) > + if (orig_resid =3D=3D zfs_uio_resid(&uio)= && > + eofp !=3D NULL) > *eofp =3D 1; > error =3D 0; > } > @@ -1099,8 +1115,12 @@ zfsctl_snapdir_readdir(struct vop_readdir_args *ap= ) > entry.d_reclen =3D sizeof (entry); > error =3D vfs_read_dirent(ap, &entry, zfs_uio_offset(&uio= )); > if (error !=3D 0) { > - if (error =3D=3D ENAMETOOLONG) > + if (error =3D=3D ENAMETOOLONG) { > error =3D 0; > + if (orig_resid =3D=3D zfs_uio_resid(&uio)= && > + eofp !=3D NULL) > + *eofp =3D 1; Ditto again. > + } > zfs_exit(zfsvfs, FTAG); > return (SET_ERROR(error)); > } > diff --git a/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c b/s= ys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c > index 1813c411b013..79e808a3ab3d 100644 > --- a/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c > +++ b/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c > @@ -1695,6 +1695,7 @@ zfs_readdir(vnode_t *vp, zfs_uio_t *uio, cred_t *cr= , int *eofp, > objset_t *os; > caddr_t outbuf; > size_t bufsize; > + ssize_t orig_resid; > zap_cursor_t zc; > zap_attribute_t *zap; > uint_t bytes_wanted; > @@ -1735,7 +1736,7 @@ zfs_readdir(vnode_t *vp, zfs_uio_t *uio, cred_t *cr= , int *eofp, > /* > * Quit if directory has been removed (posix) > */ > - if ((*eofp =3D zp->z_unlinked) !=3D 0) { > + if ((*eofp =3D (zp->z_unlinked !=3D 0)) !=3D 0) { > zfs_exit(zfsvfs, FTAG); > return (0); > } > @@ -1743,6 +1744,7 @@ zfs_readdir(vnode_t *vp, zfs_uio_t *uio, cred_t *cr= , int *eofp, > error =3D 0; > os =3D zfsvfs->z_os; > offset =3D zfs_uio_offset(uio); > + orig_resid =3D zfs_uio_resid(uio); > prefetch =3D zp->z_zn_prefetch; > zap =3D zap_attribute_long_alloc(); > > @@ -1933,6 +1935,8 @@ zfs_readdir(vnode_t *vp, zfs_uio_t *uio, cred_t *cr= , int *eofp, > *cookies =3D NULL; > *ncookies =3D 0; > } > + if (error =3D=3D 0 && zfs_uio_resid(uio) !=3D orig_resid) > + *eofp =3D 1; I looked at this earlier and I don't this is needed. When it breaks out of the loop, it has set eof if it got ENOENT from zap_cursor_retrieve(), which I think is the only case that will return 0 and not an error? (But it looks harmless to me.) > return (error); > } > >