14.3R zfs panic VERIFY3(zfs_btree_find(tree, node, &where) == NULL) failed (0xfffff80090dd93a0x == 0x)

From: mike tancsa <mike_at_sentex.net>
Date: Thu, 12 Jun 2025 11:12:25 UTC
Not sure if this is operator error or triggering a bug.   But my process 
worked fine with a 14.2R base image.

But the broad steps I am doing

download a 14.3R VM image.

start it up with bhyve, do a zpool upgrade to enable all the features 
etc. Shut it down.

copy that image to new.image.  mdconfig -f new.image, import the pool, 
add some packages, copy over some localizations etc. export the pool, 
mdconfig -d and then dd if=newimage of= msata disk.  Put msata disk in 
new box, boot up. I did this on 2 separate machines and both crashed 
with the same panic at night when periodic daily runs.

The exact burn process is

     zpool export  ${TARGETPOOL}
         mdconfig -d -u 0
         gpart destroy -F ${TARGETDEVICE}
         set -e
         dd if=/dev/zero of=${TARGETDEVICE} count=500 bs=1m
         #{BASEPOOL}/${NAME}/${BSD} = modified 14.3.raw

         dd if=/${BASEPOOL}/${NAME}/${BSD} of=${TARGETDEVICE} bs=64k 
status=progress
         cat /dev/null > ${TARGETDEVICE}
         usbconfig -d 0.3 reset
         sleep 5
         gpart recover ${TARGETDEVICE}
         #sysctl -w kern.geom.debugflags=16
         zpool import -f -R $TARGETMOUNT ${NAME}
         zpool upgrade ${NAME}
         zpool export ${NAME}


<6>1 2025-06-12T03:01:41.088606-04:00 site246.sentex.ca syslogd - - - 
kernel boot file is /boot/kernel/kernel
<2>1 2025-06-12T03:01:41.088834-04:00 site246.sentex.ca kernel - - - 
panic: VERIFY3(zfs_btree_find(tree, node, &where) == NULL) failed 
(0xfffff80090dd93a0x == 0x)
<2>1 2025-06-12T03:01:41.088921-04:00 site246.sentex.ca kernel - - -
<2>1 2025-06-12T03:01:41.089056-04:00 site246.sentex.ca kernel - - - 
cpuid = 3
<2>1 2025-06-12T03:01:41.089139-04:00 site246.sentex.ca kernel - - - 
time = 1749711662
<2>1 2025-06-12T03:01:41.089223-04:00 site246.sentex.ca kernel - - - 
KDB: stack backtrace:
<2>1 2025-06-12T03:01:41.089306-04:00 site246.sentex.ca kernel - - - #0 
0xffffffff80ba8f1d at kdb_backtrace+0x5d
<2>1 2025-06-12T03:01:41.089388-04:00 site246.sentex.ca kernel - - - #1 
0xffffffff80b5aa11 at vpanic+0x161
<2>1 2025-06-12T03:01:41.089496-04:00 site246.sentex.ca kernel - - - #2 
0xffffffff82d91c4a at spl_panic+0x3a
<2>1 2025-06-12T03:01:41.089578-04:00 site246.sentex.ca kernel - - - #3 
0xffffffff82de4aae at zfs_btree_add+0x6e
<2>1 2025-06-12T03:01:41.089661-04:00 site246.sentex.ca kernel - - - #4 
0xffffffff82f25905 at zap_lockdir_impl+0x325
<2>1 2025-06-12T03:01:41.089743-04:00 site246.sentex.ca kernel - - - #5 
0xffffffff82f255a4 at zap_lockdir+0x84
<2>1 2025-06-12T03:01:41.089825-04:00 site246.sentex.ca kernel - - - #6 
0xffffffff82f28b15 at zap_cursor_retrieve+0x175
<2>1 2025-06-12T03:01:41.089930-04:00 site246.sentex.ca kernel - - - #7 
0xffffffff82db01a3 at zfs_freebsd_readdir+0x383
<2>1 2025-06-12T03:01:41.090014-04:00 site246.sentex.ca kernel - - - #8 
0xffffffff81112050 at VOP_READDIR_APV+0x20
<2>1 2025-06-12T03:01:41.090096-04:00 site246.sentex.ca kernel - - - #9 
0xffffffff80c56a19 at kern_getdirentries+0x229
<2>1 2025-06-12T03:01:41.090178-04:00 site246.sentex.ca kernel - - - #10 
0xffffffff80c56dc9 at sys_getdirentries+0x29
<2>1 2025-06-12T03:01:41.090278-04:00 site246.sentex.ca kernel - - - #11 
0xffffffff8104e4c7 at amd64_syscall+0x117
<2>1 2025-06-12T03:01:41.090361-04:00 site246.sentex.ca kernel - - - 
---<<BOOT>>---



<46>1 2025-06-12T03:01:37.885430-04:00 site724.sentex.ca syslogd - - - 
restart
<6>1 2025-06-12T03:01:37.886599-04:00 site724.sentex.ca syslogd - - - 
kernel boot file is /boot/kernel/kernel
<2>1 2025-06-12T03:01:37.886786-04:00 site724.sentex.ca kernel - - - 
panic: VERIFY3(zfs_btree_find(tree, node, &where) == NULL) failed 
(0xfffff8002e594ba0x == 0x)
<2>1 2025-06-12T03:01:37.886881-04:00 site724.sentex.ca kernel - - -
<2>1 2025-06-12T03:01:37.886964-04:00 site724.sentex.ca kernel - - - 
cpuid = 1
<2>1 2025-06-12T03:01:37.887047-04:00 site724.sentex.ca kernel - - - 
time = 1749711662
<2>1 2025-06-12T03:01:37.887167-04:00 site724.sentex.ca kernel - - - 
KDB: stack backtrace:
<2>1 2025-06-12T03:01:37.887252-04:00 site724.sentex.ca kernel - - - #0 
0xffffffff80ba8f1d at kdb_backtrace+0x5d
<2>1 2025-06-12T03:01:37.887343-04:00 site724.sentex.ca kernel - - - #1 
0xffffffff80b5aa11 at vpanic+0x161
<2>1 2025-06-12T03:01:37.887426-04:00 site724.sentex.ca kernel - - - #2 
0xffffffff82d91c4a at spl_panic+0x3a
<2>1 2025-06-12T03:01:37.887509-04:00 site724.sentex.ca kernel - - - #3 
0xffffffff82de4aae at zfs_btree_add+0x6e
<2>1 2025-06-12T03:01:37.887611-04:00 site724.sentex.ca kernel - - - #4 
0xffffffff82f25905 at zap_lockdir_impl+0x325
<2>1 2025-06-12T03:01:37.887694-04:00 site724.sentex.ca kernel - - - #5 
0xffffffff82f255a4 at zap_lockdir+0x84
<2>1 2025-06-12T03:01:37.887777-04:00 site724.sentex.ca kernel - - - #6 
0xffffffff82f28b15 at zap_cursor_retrieve+0x175
<2>1 2025-06-12T03:01:37.887866-04:00 site724.sentex.ca kernel - - - #7 
0xffffffff82db01a3 at zfs_freebsd_readdir+0x383
<2>1 2025-06-12T03:01:37.887954-04:00 site724.sentex.ca kernel - - - #8 
0xffffffff81112050 at VOP_READDIR_APV+0x20
<2>1 2025-06-12T03:01:37.888054-04:00 site724.sentex.ca kernel - - - #9 
0xffffffff80c56a19 at kern_getdirentries+0x229
<2>1 2025-06-12T03:01:37.888138-04:00 site724.sentex.ca kernel - - - #10 
0xffffffff80c56dc9 at sys_getdirentries+0x29
<2>1 2025-06-12T03:01:37.888221-04:00 site724.sentex.ca kernel - - - #11 
0xffffffff8104e4c7 at amd64_syscall+0x117
<2>1 2025-06-12T03:01:37.888304-04:00 site724.sentex.ca kernel - - - 
---<<BOOT>>---

running periodic daily manually can trigger the panic on both systems.  
scrub does not show any errors


# zpool status
   pool: site724
  state: ONLINE
   scan: scrub repaired 0B in 00:00:18 with 0 errors on Thu Jun 12 
07:06:55 2025
config:

         NAME          STATE     READ WRITE CKSUM
         site724      ONLINE       0     0     0
           gpt/rootfs  ONLINE       0     0     0

errors: No known data errors

  # periodic daily
.... and 2 or 3 seconds later, the panic.