14.3R zfs panic VERIFY3(zfs_btree_find(tree, node, &where) == NULL) failed (0xfffff80090dd93a0x == 0x)
- Reply: mike tancsa : "Re: 14.3R zfs panic VERIFY3(zfs_btree_find(tree, node, &where) == NULL) failed (0xfffff80090dd93a0x == 0x)"
- Reply: mike tancsa : "Re: 14.3R zfs panic VERIFY3(zfs_btree_find(tree, node, &where) == NULL) failed (0xfffff80090dd93a0x == 0x)"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Thu, 12 Jun 2025 11:12:25 UTC
Not sure if this is operator error or triggering a bug. But my process
worked fine with a 14.2R base image.
But the broad steps I am doing
download a 14.3R VM image.
start it up with bhyve, do a zpool upgrade to enable all the features
etc. Shut it down.
copy that image to new.image. mdconfig -f new.image, import the pool,
add some packages, copy over some localizations etc. export the pool,
mdconfig -d and then dd if=newimage of= msata disk. Put msata disk in
new box, boot up. I did this on 2 separate machines and both crashed
with the same panic at night when periodic daily runs.
The exact burn process is
zpool export ${TARGETPOOL}
mdconfig -d -u 0
gpart destroy -F ${TARGETDEVICE}
set -e
dd if=/dev/zero of=${TARGETDEVICE} count=500 bs=1m
#{BASEPOOL}/${NAME}/${BSD} = modified 14.3.raw
dd if=/${BASEPOOL}/${NAME}/${BSD} of=${TARGETDEVICE} bs=64k
status=progress
cat /dev/null > ${TARGETDEVICE}
usbconfig -d 0.3 reset
sleep 5
gpart recover ${TARGETDEVICE}
#sysctl -w kern.geom.debugflags=16
zpool import -f -R $TARGETMOUNT ${NAME}
zpool upgrade ${NAME}
zpool export ${NAME}
<6>1 2025-06-12T03:01:41.088606-04:00 site246.sentex.ca syslogd - - -
kernel boot file is /boot/kernel/kernel
<2>1 2025-06-12T03:01:41.088834-04:00 site246.sentex.ca kernel - - -
panic: VERIFY3(zfs_btree_find(tree, node, &where) == NULL) failed
(0xfffff80090dd93a0x == 0x)
<2>1 2025-06-12T03:01:41.088921-04:00 site246.sentex.ca kernel - - -
<2>1 2025-06-12T03:01:41.089056-04:00 site246.sentex.ca kernel - - -
cpuid = 3
<2>1 2025-06-12T03:01:41.089139-04:00 site246.sentex.ca kernel - - -
time = 1749711662
<2>1 2025-06-12T03:01:41.089223-04:00 site246.sentex.ca kernel - - -
KDB: stack backtrace:
<2>1 2025-06-12T03:01:41.089306-04:00 site246.sentex.ca kernel - - - #0
0xffffffff80ba8f1d at kdb_backtrace+0x5d
<2>1 2025-06-12T03:01:41.089388-04:00 site246.sentex.ca kernel - - - #1
0xffffffff80b5aa11 at vpanic+0x161
<2>1 2025-06-12T03:01:41.089496-04:00 site246.sentex.ca kernel - - - #2
0xffffffff82d91c4a at spl_panic+0x3a
<2>1 2025-06-12T03:01:41.089578-04:00 site246.sentex.ca kernel - - - #3
0xffffffff82de4aae at zfs_btree_add+0x6e
<2>1 2025-06-12T03:01:41.089661-04:00 site246.sentex.ca kernel - - - #4
0xffffffff82f25905 at zap_lockdir_impl+0x325
<2>1 2025-06-12T03:01:41.089743-04:00 site246.sentex.ca kernel - - - #5
0xffffffff82f255a4 at zap_lockdir+0x84
<2>1 2025-06-12T03:01:41.089825-04:00 site246.sentex.ca kernel - - - #6
0xffffffff82f28b15 at zap_cursor_retrieve+0x175
<2>1 2025-06-12T03:01:41.089930-04:00 site246.sentex.ca kernel - - - #7
0xffffffff82db01a3 at zfs_freebsd_readdir+0x383
<2>1 2025-06-12T03:01:41.090014-04:00 site246.sentex.ca kernel - - - #8
0xffffffff81112050 at VOP_READDIR_APV+0x20
<2>1 2025-06-12T03:01:41.090096-04:00 site246.sentex.ca kernel - - - #9
0xffffffff80c56a19 at kern_getdirentries+0x229
<2>1 2025-06-12T03:01:41.090178-04:00 site246.sentex.ca kernel - - - #10
0xffffffff80c56dc9 at sys_getdirentries+0x29
<2>1 2025-06-12T03:01:41.090278-04:00 site246.sentex.ca kernel - - - #11
0xffffffff8104e4c7 at amd64_syscall+0x117
<2>1 2025-06-12T03:01:41.090361-04:00 site246.sentex.ca kernel - - -
---<<BOOT>>---
<46>1 2025-06-12T03:01:37.885430-04:00 site724.sentex.ca syslogd - - -
restart
<6>1 2025-06-12T03:01:37.886599-04:00 site724.sentex.ca syslogd - - -
kernel boot file is /boot/kernel/kernel
<2>1 2025-06-12T03:01:37.886786-04:00 site724.sentex.ca kernel - - -
panic: VERIFY3(zfs_btree_find(tree, node, &where) == NULL) failed
(0xfffff8002e594ba0x == 0x)
<2>1 2025-06-12T03:01:37.886881-04:00 site724.sentex.ca kernel - - -
<2>1 2025-06-12T03:01:37.886964-04:00 site724.sentex.ca kernel - - -
cpuid = 1
<2>1 2025-06-12T03:01:37.887047-04:00 site724.sentex.ca kernel - - -
time = 1749711662
<2>1 2025-06-12T03:01:37.887167-04:00 site724.sentex.ca kernel - - -
KDB: stack backtrace:
<2>1 2025-06-12T03:01:37.887252-04:00 site724.sentex.ca kernel - - - #0
0xffffffff80ba8f1d at kdb_backtrace+0x5d
<2>1 2025-06-12T03:01:37.887343-04:00 site724.sentex.ca kernel - - - #1
0xffffffff80b5aa11 at vpanic+0x161
<2>1 2025-06-12T03:01:37.887426-04:00 site724.sentex.ca kernel - - - #2
0xffffffff82d91c4a at spl_panic+0x3a
<2>1 2025-06-12T03:01:37.887509-04:00 site724.sentex.ca kernel - - - #3
0xffffffff82de4aae at zfs_btree_add+0x6e
<2>1 2025-06-12T03:01:37.887611-04:00 site724.sentex.ca kernel - - - #4
0xffffffff82f25905 at zap_lockdir_impl+0x325
<2>1 2025-06-12T03:01:37.887694-04:00 site724.sentex.ca kernel - - - #5
0xffffffff82f255a4 at zap_lockdir+0x84
<2>1 2025-06-12T03:01:37.887777-04:00 site724.sentex.ca kernel - - - #6
0xffffffff82f28b15 at zap_cursor_retrieve+0x175
<2>1 2025-06-12T03:01:37.887866-04:00 site724.sentex.ca kernel - - - #7
0xffffffff82db01a3 at zfs_freebsd_readdir+0x383
<2>1 2025-06-12T03:01:37.887954-04:00 site724.sentex.ca kernel - - - #8
0xffffffff81112050 at VOP_READDIR_APV+0x20
<2>1 2025-06-12T03:01:37.888054-04:00 site724.sentex.ca kernel - - - #9
0xffffffff80c56a19 at kern_getdirentries+0x229
<2>1 2025-06-12T03:01:37.888138-04:00 site724.sentex.ca kernel - - - #10
0xffffffff80c56dc9 at sys_getdirentries+0x29
<2>1 2025-06-12T03:01:37.888221-04:00 site724.sentex.ca kernel - - - #11
0xffffffff8104e4c7 at amd64_syscall+0x117
<2>1 2025-06-12T03:01:37.888304-04:00 site724.sentex.ca kernel - - -
---<<BOOT>>---
running periodic daily manually can trigger the panic on both systems.
scrub does not show any errors
# zpool status
pool: site724
state: ONLINE
scan: scrub repaired 0B in 00:00:18 with 0 errors on Thu Jun 12
07:06:55 2025
config:
NAME STATE READ WRITE CKSUM
site724 ONLINE 0 0 0
gpt/rootfs ONLINE 0 0 0
errors: No known data errors
# periodic daily
.... and 2 or 3 seconds later, the panic.