freebsd-update 12.3 to 14.0RC1 takes 12-24 hours (block cloning regression)

From: Kevin Bowling <kevin.bowling_at_kev009.com>
Date: Tue, 17 Oct 2023 16:40:37 UTC
Hi,

I have two systems with a zpool 2x2 mirror on 7.2k RPM disks.  One
system also has a flash SLOG.

The flash SLOG system took around 12 hours to complete freebsd-update
from 13.2 to 14.0-RC1.  The system without the SLOG took nearly 24
hours.  This was the result of ~50k patches, and ~10k files from
freebsd-update and a very pathological 'install' command performance.

'ps auxww | grep install':
root   52225   0.0  0.0  12852   2504  0  D+   20:55      0:00.00
install -S -o 0 -g 0 -m 0644
b6850914127c27fe192a41387f5cec04a1d927e6605ff09e8fd88dcd74fdec9d
///usr/src/sys/netgraph/ng_vlan.h
root   68042   0.0  0.0  13580   3648  0  I+   02:24      0:01.14
/bin/sh /usr/sbin/freebsd-update install              root   69946
0.0  0.0  13580   3632  0  S+   02:24      0:15.65 /bin/sh
/usr/sbin/freebsd-update install

'control+t on freebsd-update':

load: 0.16  cmd: install 97128 [tx->tx_sync_done_cv] 0.67r 0.00u 0.00s
0% 2440k
mi_switch+0xc2 _cv_wait+0x113 txg_wait_synced_impl+0xb9
txg_wait_synced+0xb dmu_offset_next+0x77 zfs_holey+0x137 zfs_fre
ebsd_ioctl+0x4f vn_generic_copy_file_range+0x64b
kern_copy_file_range+0x327 sys_copy_file_range+0x78
amd64_syscall+0x10c
 fast_syscall_common+0xf8

I spoke with mjg about this and because my pools do not have block
cloning enabled, copy_file_range turns into a massive pessimization in
'install'.  He suggested a workaround of 'sysctl
vfs.zfs.dmu_offset_next_sync=0' but we should probably sort this out
for 14.0-RELEASE.

Regards,
Kevin