Re: Is 14.0 to released based on 0 for sysctl vfs.zfs.bclone_enabled ?

On Fri, Nov 10, 2023 at 7:50 AM Martin Matuska <mm@freebsd.org> wrote:

> Hi Xin,
>
> since when have you been using block cloning on the system? Is it
> possible that there is already corrupted block-cloned data from the


That's a good question, I can't 100% rule out this possibility.  I was
following -CURRENT in ~weekly to ~monthly on that system, and the pool was
created in March 2014.

Do you think I should try rebuilding the pool from scratch?  I do have
remote backup on a different server but was avoiding it because it's time
consuming.


> past? Is everything on one dataset or are you using multiple datasets
> for /usr/src and /usr/obj?
>

/usr/src and /usr/obj are separate datasets, and the system runs Poudriere
so it have multiple copies of slightly different /usr/src and /usr/obj's.

Is there a way to identify datasets with block cloning, by the way?  Maybe
I should try recreating these datasets first?



>
> Best regards,
> mm
>
> On 10. 11. 2023 8:04, Xin Li wrote:
> > On 2023-11-05 16:34, Martin Matuska wrote:
> >> OpenZFS 2.2.0 in FreeBSD 14 fully supports block cloning. You can
> >> work with pools that have feature@block_cloning enabled.
> >> The sysctl variable vfs.zfs.bclone_enabled affects the behavior of
> >> zfs_clone_range() which is called by copy_file_range(). When it is
> >> set to 0, zfs_clone_range() does not do block cloning.
> >> If it is set to anything else than 0, zfs_clone_range() does block
> >> cloning (if all conditions are met - same ZFS pool, correct data
> >> alignment, etc.).
> >>
> >> In FreeBSD-main, this tunable is enabled and I plan to enable it in
> >> stable/14 somewhere around December 11, 2023.
> >>
> >> As of today I personally use block cloning on all my systems.
> >
> > I'd like to share a different data point.  It still panics on my
> > storage (running -CURRENT about a week ago) when enabled and can be
> > triggered by "make buildworld buildkernel".  I wasn't able to capture
> > earlier coredump until the most recent one, which panicked with:
> >
> >
> > cpuid = 2
> > time = 1699593456
> > KDB: stack backtrace:
> > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> > 0xfffffe022f2bd7e0
> > vpanic() at vpanic+0x132/frame 0xfffffe022f2bd910
> > spl_panic() at spl_panic+0x3a/frame 0xfffffe022f2bd970
> > dmu_brt_clone() at dmu_brt_clone+0x555/frame 0xfffffe022f2bd9e0
> > zfs_clone_range() at zfs_clone_range+0xa4c/frame 0xfffffe022f2bdbb0
> > zfs_freebsd_copy_file_range() at
> > zfs_freebsd_copy_file_range+0x18a/frame 0xfffffe022f2bdc30
> > vn_copy_file_range() at vn_copy_file_range+0x163/frame 0xfffffe022f2bdce0
> > kern_copy_file_range() at kern_copy_file_range+0x380/frame
> > 0xfffffe022f2bddb0
> > sys_copy_file_range() at sys_copy_file_range+0x78/frame
> > 0xfffffe022f2bde00
> > amd64_syscall() at amd64_syscall+0x153/frame 0xfffffe022f2bdf30
> > fast_syscall_common() at fast_syscall_common+0xf8/frame
> > 0xfffffe022f2bdf30
> > --- syscall (569, FreeBSD ELF64, copy_file_range), rip =
> > 0x7fbb2da4ada, rsp = 0x7fbb02c5d48, rbp = 0x7fbb02c61e0 ---
> > Uptime: 2h32m27s
> > Dumping 7800 out of 32696
> > MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
> >
> > #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
> > #1  doadump (textdump=textdump@entry=1) at
> > /usr/src/sys/kern/kern_shutdown.c:405
> > #2  0xffffffff80694480 in kern_reboot (howto=260) at
> > /usr/src/sys/kern/kern_shutdown.c:526
> > #3  0xffffffff8069497f in vpanic (fmt=0xffffffff82603415 "VERIFY3(nbps
> > == numbufs) failed (%llu == %llu)\n", ap=ap@entry=0xfffffe022f2bd950)
> > at /usr/src/sys/kern/kern_shutdown.c:970
> > #4  0xffffffff8232999a in spl_panic (file=<optimized out>,
> > func=<optimized out>, line=<unavailable>, fmt=<unavailable>) at
> > /usr/src/sys/contrib/openzfs/module/os/freebsd/spl/spl_misc.c:103
> > #5  0xffffffff823a6605 in dmu_brt_clone
> > (os=os@entry=0xfffff800c5ce4000, object=<optimized out>,
> > offset=offset@entry=0, length=length@entry=207477,
> > tx=tx@entry=0xfffff8071a108d00, bps=bps@entry=0xfffffe01e218c000,
> > nbps=2, replay=0)
> >     at /usr/src/sys/contrib/openzfs/module/zfs/dmu.c:2303
> > #6  0xffffffff8250f67c in zfs_clone_range (inzp=0xfffff804416ac000,
> > inoffp=0xfffff800b81cb048, outzp=0xfffff806f58f03a0,
> > outoffp=0xfffff800b8063048, lenp=lenp@entry=0xfffffe022f2bdbf0,
> > cr=0xfffff8000a6fe600)
> >     at /usr/src/sys/contrib/openzfs/module/zfs/zfs_vnops.c:1326
> > #7  0xffffffff8234b3ba in zfs_freebsd_copy_file_range
> > (ap=0xfffffe022f2bdc48) at
> > /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:6294
> > #8  0xffffffff8079f443 in VOP_COPY_FILE_RANGE
> > (invp=0xfffff804416cb1c0, inoffp=0xfffff800b81cb048,
> > outvp=0xfffff806f51d3380, outoffp=0xfffff800b8063048,
> > lenp=0xfffffe022f2bdd48, incred=0xfffff8000a6fe600, flags=<optimized
> > out>,
> >     outcred=<optimized out>, fsizetd=<optimized out>) at
> > ./vnode_if.h:2385
> > #9  vn_copy_file_range (invp=invp@entry=0xfffff804416cb1c0,
> > inoffp=inoffp@entry=0xfffff800b81cb048,
> > outvp=outvp@entry=0xfffff806f51d3380,
> > outoffp=outoffp@entry=0xfffff800b8063048,
> > lenp=lenp@entry=0xfffffe022f2bdd48, flags=flags@entry=0,
> >     incred=0xfffff8000a6fe600, outcred=0xfffff8000a6fe600,
> > fsize_td=0xfffffe022925b3a0) at /usr/src/sys/kern/vfs_vnops.c:3087
> > #10 0xffffffff8079a070 in kern_copy_file_range
> > (td=td@entry=0xfffffe022925b3a0, infd=<optimized out>,
> > inoffp=0xfffff800b81cb048, inoffp@entry=0x0, outfd=<optimized out>,
> > outoffp=0xfffff800b8063048, outoffp@entry=0x0, len=9223372036854775807,
> >     flags=0) at /usr/src/sys/kern/vfs_syscalls.c:4973
> > #11 0xffffffff8079a178 in sys_copy_file_range (td=0xfffffe022925b3a0,
> > uap=0xfffffe022925b7a0) at /usr/src/sys/kern/vfs_syscalls.c:5011
> > #12 0xffffffff80a97aa3 in syscallenter (td=0xfffffe022925b3a0) at
> > /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:188
> > #13 amd64_syscall (td=0xfffffe022925b3a0, traced=0) at
> > /usr/src/sys/amd64/amd64/trap.c:1194
> > #14 <signal handler called>
> > #15 0x000007fbb2da4ada in ?? ()
> >
> >
> > and disabling bclone does appear to allow me to finish buildworld /
> > buildkernel.
> >
> > The pool didn't have redaction_list_spill enabled.
> >
> > The ASSERT3U(nbps, ==, numbufs); in dmu_brt_clone was added when block
> > clone is first implemented.
> >
> > It seems that I am the only person who is seeing this as of today.  It
> > seems that block clone was indeed being used for some data:
> >
> > saturn  bcloneused 1.18M                          -
> > saturn  bclonesaved 1.21M                          -
> > saturn  bcloneratio 2.02x                          -
> >
> > The pool have dedup enabled for some datasets.
> >
> > Any suggestions?  (In extreme cases I can recreate the storage pool
> > from backup or copy the data somewhere else, then recreate the pool,
> > then copy data back, but I'd like to avoid that if possible)
> >
> > Cheers,
> >
> >
> >>
> >> mm
> >>
> >> On 04/11/2023 13:35, Mark Millard wrote:
> >>> On Nov 4, 2023, at 04:38, Mike Karels <mike@karels.net> wrote:
> >>>
> >>>> On 4 Nov 2023, at 4:01, Ronald Klop wrote:
> >>>>
> >>>>> On 11/4/23 02:39, Mark Millard wrote:
> >>>>>> It looks to me like releng/14.0 (as of 14.0-RC4) still has:
> >>>>>>
> >>>>>> int zfs_bclone_enabled;
> >>>>>> SYSCTL_INT(_vfs_zfs, OID_AUTO, bclone_enabled, CTLFLAG_RWTUN,
> >>>>>> &zfs_bclone_enabled, 0, "Enable block cloning");
> >>>>>>
> >>>>>> leaving block cloning effectively disabled by default, no
> >>>>>> matter what the pool has enabled.
> >>>>>>
> >>>>>> https://www.freebsd.org/releases/14.0R/relnotes/ also reports:
> >>>>>>
> >>>>>> QUOTE
> >>>>>> OpenZFS has been upgraded to version 2.2. New features include:
> >>>>>> •
> >>>>>> block cloning, which allows shallow copies of blocks in file
> >>>>>> copies. This is optional, and disabled by default; it can be
> >>>>>> enabled with sysctl vfs.zfs.bclone_enabled=1.
> >>>>>> END QUOTE
> >>>>>>
> >>>>>
> >>>>> I think this answers your question in the subject.
> >>>> I think so too (and I wrote that text).
> >>> Thanks for the confirmation of the final intent.
> >>>
> >>> I believe this makes:
> >>>
> >>> QUOTE
> >>> author Brian Behlendorf <behlendorf1@llnl.gov> 2023-05-25 20:53:08
> >>> +0000
> >>> committer GitHub <noreply@github.com> 2023-05-25 20:53:08 +0000
> >>> commit 91a2325c4a0fbe01d0bf212e44fa9d85017837ce (patch)
> >>> tree dd01dfce6aeef357ade1775acf18aade535c6271
> >>> . . .
> >>> Update compatibility.d files
> >>>
> >>> Add an openzfs-2.2 compatibility file for the next release. Edon-R
> >>> support has been enabled for FreeBSD removing the need for different
> >>> FreeBSD and Linux files. Symlinks for the -linux and -freebsd names
> >>> are created for any scripts expecting that convention. Additionally,
> >>> a symlink for ubunutu-22.04 was added. Signed-off-by: Brian
> >>> Behlendorf <behlendorf1@llnl.gov> Closes #14833
> >>> END QUOTE
> >>>
> >>> technically incorrect in that compatibility.d/openzfs-2.2-freebsd
> >>> should be distinct in content from compatibility.d/openzfs-2.2 so
> >>> that block cloning would not be enabled.
> >>>
> >>>
> >>>>>> Just curiousity on my part about the default completeness of
> >>>>>> openzfs-2.2 support, not an objection either way.
> >>>>>>
> >>>>>
> >>>>> I haven't seen new issues with block cloning in the last few weeks
> >>>>> mentioned on the mailing lists. All known issues are fixed AFAIK.
> >>>>> But I can imagine that the risk+effect ratio of data corruption is
> >>>>> seen as a bit too high for a 14.0 release for this particular
> >>>>> feature. That does not diminish the rest of the completeness of
> >>>>> openzfs-2.2.
> >>>>>
> >>>>> NB: I'm not involved in developing openzfs or the decision making
> >>>>> in the release. Just repeating what I read on the lists.
> >>>> There was another block cloning fix in 14.0-RC4; see the commit log.
> >>>> Maybe there will be no more issues, but it seems that corner cases
> >>>> were
> >>>> still being found recently.
> >>> Looks like I'll stay at openzfs-2.1 pool features until there is
> >>> a release that no longer has the default status:
> >>>
> >>> 0 for sysctl vfs.zfs.bclone_enabled
> >>>
> >>> I use main [so: 15 now] but only enable openzfs-2.* pool features
> >>> supported by default on some FreeBSD release, that has an accurate
> >>> compatibility.d/openzfs-2.*-freebsd file.
> >>>
> >>> ===
> >>> Mark Millard
> >>> marklmi at yahoo.com
> >>>
> >>>
> >>
> >
>
>