Re: 14.3R zfs panic VERIFY3(zfs_btree_find(tree, node, &where) == NULL) failed (0xfffff80090dd93a0x == 0x)
Date: Thu, 12 Jun 2025 15:37:49 UTC
On 6/12/2025 7:12 AM, mike tancsa wrote:
> Not sure if this is operator error or triggering a bug. But my
> process worked fine with a 14.2R base image.
>
> running periodic daily manually can trigger the panic on both
> systems. scrub does not show any errors
>
I simplified the steps to recreate the issue.
0{july04apu}# fetch
https://download.freebsd.org/releases/VM-IMAGES/14.3-RELEASE/amd64/Latest/FreeBSD-14.3-RELEASE-amd64-zfs.raw.xz
FreeBSD-14.3-RELEASE-amd64-zfs.raw.xz 812 MB 9623 kBps
01m26s
0{july04apu}# sha256 FreeBSD-14.3-RELEASE-amd64-zfs.raw.xz
SHA256 (FreeBSD-14.3-RELEASE-amd64-zfs.raw.xz) =
0a1a861763b30d9bdaf3d011a6252d6a63d52cf0db94edad4319af2609fd2791
0{july04apu}# xz -d FreeBSD-14.3-RELEASE-amd64-zfs.raw.xz
0{july04apu}# sha256 FreeBSD-14.3-RELEASE-amd64-zfs.raw
SHA256 (FreeBSD-14.3-RELEASE-amd64-zfs.raw) =
fd171dd5afcaed30102bdf604d54dc26547e1ece73fa7a5be81c684438880232
0{july04apu}# cp FreeBSD-14.3-RELEASE-amd64-zfs.raw 143.raw
0{july04apu}# sha256 143.raw
SHA256 (143.raw) =
fd171dd5afcaed30102bdf604d54dc26547e1ece73fa7a5be81c684438880232
0{july04apu}# vmrun.sh -t tap0 -d ./143.raw -E vm0
Launching virtual machine "vm0" ...
fbuf frame buffer base: 0x252ca7200000 [sz 16777216]
3h
3h
3h
BdsDxe: loading Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x3,0x0)
BdsDxe: starting Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x3,0x0)
Consoles: EFI console
Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk0p2:
FreeBSD/amd64 EFI loader, Revision 3.0
Command line arguments: loader.efi
Image base: 0x1e27a000
EFI version: 2.70
EFI Firmware: BHYVE (rev 1.00)
Console: efi (0x20001000)
Load Path: \EFI\BOOT\BOOTX64.EFI
Load Device:
PciRoot(0x0)/Pci(0x3,0x0)/HD(2,GPT,A09CDCFB-42D1-11F0-9E0B-002590EC5BF2,0x17B,0x10418)
BootCurrent: 0001
BootOrder: 0000 0001[*] 0002 0003 0004
BootInfo Path: PciRoot(0x0)/Pci(0x3,0x0)
Ignoring Boot0001: Only one DP found
Trying ESP:
PciRoot(0x0)/Pci(0x3,0x0)/HD(2,GPT,A09CDCFB-42D1-11F0-9E0B-002590EC5BF2,0x17B,0x10418)
Setting currdev to disk0p2:
Trying:
PciRoot(0x0)/Pci(0x3,0x0)/HD(1,GPT,A09CDCEF-42D1-11F0-9E0B-002590EC5BF2,0x22,0x159)
Setting currdev to disk0p1:
Trying:
PciRoot(0x0)/Pci(0x3,0x0)/HD(3,GPT,A09CDD17-42D1-11F0-9E0B-002590EC5BF2,0x10593,0x200000)
Setting currdev to disk0p3:
Trying:
PciRoot(0x0)/Pci(0x3,0x0)/HD(4,GPT,A09CDD1B-42D1-11F0-9E0B-002590EC5BF2,0x210593,0xA00000)
Setting currdev to zfs:zroot/ROOT/default:
\
Loading /boot/defaults/loader.conf
Loading /boot/defaults/loader.conf
Loading /boot/device.hints
Loading /boot/loader.conf
Loading /boot/loader.conf.local
?c-
\ ______ ____ _____ _____
| ____| | _ \ / ____| __ \
| |___ _ __ ___ ___ | |_) | (___ | | | |
| ___| '__/ _ \/ _ \| _ < \___ \| | | |
| | | | | __/ __/| |_) |____) | |__| |
| | | | | | || | | |
|_| |_| \___|\___||____/|_____/|_____/ ``` `
s`
`.....---.......--.``` -/
3;02H9;44H3;44H9;03H3;03H0;02H0;44H1;02H1;44H2;02H2;44H3;02H3;44H4;02H4;44H5;02H5;44H6;02H6;44H7;02H7;44H8;02H8;44H9;02H9;44H0;02H0;44H1;02H1;44H2;02H2;44H9;13H
Welcome to FreeBSD
yo`:. :o `+-
1. Boot Multi user [Enter] y/ -/` -o/
2. Boot Single user .- ::/sy+:.
3. Escape to loader prompt / `-- /
4. Reboot `: :`
5. Cons: Dual (Serial primary) `: :`
/ /
Options: .- -.
6. Kernel: default/kernel (1 of 1) -- -.
7. Boot Options `:` `:`
.-- `--.
.---.....----.
Autoboot in 9 seconds. [Space] to pause
Loading kernel...
/boot/kernel/kernel text=0x180078 text=0xd97948 text=0x43a46b
data=0x180+0xe80 data=0x187590+0x478a70 0x8+0x18e940+0x8+0x1b2b03|
Loading configured modules...
/boot/kernel/zfs.ko size 0x5e9340 at 0x2142000
can't find '/etc/hostid'
can't find '/boot/entropy'
staging 0x14000000 (not copying) tramp 0x1dd83000 PT4 0x1dd7a000
Start @ 0xffffffff80381000 ...
EFI framebuffer information:
addr, size 0xc0000000, 0x1000000
dimensions 1024 x 768
stride 1024
masks 0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000
---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.3-RELEASE releng/14.3-n271432-8c9ce319fef7 GENERIC amd64
FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git
llvmorg-19.1.7-0-gcd708029e0b2)
VT(efifb): resolution 1024x768
CPU: Intel(R) N100 (800.00-MHz K8-class CPU)
Origin="GenuineIntel" Id=0xb06e0 Family=0x6 Model=0xbe Stepping=0
Features=0x9f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT,PBE>
Features2=0xfeda7a17<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,SDBG,FMA,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended
Features=0x201407a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,SMAP,SHA>
Structured Extended Features2=0x400600<VAES,VPCLMULQDQ,RDPID>
Structured Extended Features3=0x400<MD_CLEAR>
XSAVE Features=0x1<XSAVEOPT>
TSC: P-state invariant
Hypervisor: Origin = "bhyve bhyve "
real memory = 536870912 (512 MB)
avail memory = 469229568 (447 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <BHYVE BVAPIC >
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
WARNING: L2 data cache covers fewer APIC IDs than a core (0 < 1)
WARNING: L3 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 1 core(s) x 2 hardware threads
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
arc4random: WARNING: initial seeding bypassed the cryptographic random
device because it was not yet seeded and the knob
'bypass_before_seeding' was enabled.
ioapic0 <Version 1.1> irqs 0-31
Launching APs: 1
random: entropy device external interface
kbd1 at kbdmux0
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
smbios0: <System Management BIOS> at iomem 0x1fbd0000-0x1fbd001e
smbios0: Entry point: v2.1 (32-bit), Version: 2.8, BCD Revision: 2.8
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
acpi0: <BHYVE BVFACP >
acpi0: Power Button (fixed)
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 16777216 Hz quality 950
Event timer "HPET" frequency 16777216 Hz quality 550
Event timer "HPET1" frequency 16777216 Hz quality 450
Event timer "HPET2" frequency 16777216 Hz quality 450
Event timer "HPET3" frequency 16777216 Hz quality 450
Event timer "HPET4" frequency 16777216 Hz quality 450
Event timer "HPET5" frequency 16777216 Hz quality 450
Event timer "HPET6" frequency 16777216 Hz quality 450
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
virtio_pci0: <VirtIO PCI (legacy) Network adapter> port 0x2080-0x20bf
mem 0xc1002000-0xc1003fff irq 23 at device 2.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
vtnet0: Ethernet address: 00:a0:98:f0:55:05
vtnet0: netmap queues/slots: TX 1/1024, RX 1/512
000.000185 [ 452] vtnet_netmap_attach vtnet attached txq=1,
txd=1024 rxq=1, rxd=512
virtio_pci1: <VirtIO PCI (legacy) Block adapter> port 0x2000-0x207f mem
0xc1000000-0xc1001fff irq 16 at device 3.0 on pci0
vtblk0: <VirtIO Block Adapter> on virtio_pci1
vtblk0: 6176MB (12649908 512 byte sectors)
vgapci0: <VGA-compatible display> mem
0xc1004000-0xc100407f,0xc0000000-0xc0ffffff at device 29.0 on pci0
vgapci0: Boot video device
vmgenc0: <VM Generation Counter> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
driver bug: Unable to set devclass (class: atkbdc devname: (unknown))
Unhandled ps2 mouse command 0xe1
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD
15.0.
psm0: model Generic PS/2 mouse, device ID 0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart2: <16550 or compatible> port 0x3e8-0x3ef irq 4 on acpi0
uart3: <16550 or compatible> port 0x2e8-0x2ef irq 3 on acpi0
vga0: <Generic ISA VGA> at port 0x3b0-0x3bb iomem 0xb0000-0xb7fff pnpid
PNP0900 on isa0
Timecounter "TSC" frequency 806401559 Hz quality 1000
Timecounters tick every 10.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
usb_needs_explore_all: no devclass
random: unblocking device.
Trying to mount root from zfs:zroot/ROOT/default []...
Dual Console: Serial Primary, Video Secondary
Setting hostuuid: c0ccb761-1de8-f937-adf4-622a270df20f.
Setting hostid: 0x4de5a6b7.
This system supports ZFS pool feature flags.
Enabled the following features on 'zroot':
async_destroy
empty_bpobj
lz4_compress
multi_vdev_crash_dump
spacemap_histogram
enabled_txg
hole_birth
extensible_dataset
embedded_data
bookmarks
filesystem_limits
large_blocks
large_dnode
sha512
skein
edonr
userobj_accounting
encryption
project_quota
device_removal
obsolete_counts
zpool_checkpoint
spacemap_v2
allocation_classes
resilver_defer
bookmark_v2
redaction_bookmarks
redacted_datasets
bookmark_written
log_spacemap
livelist
device_rebuild
zstd_compress
draid
zilsaxattr
head_errlog
blake3
block_cloning
vdev_zaps_v2
Pool 'zroot' has the bootfs property set, you might need to update
the boot code. See gptzfsboot(8) and loader.efi(8) for details.
Starting file system checks:
/dev/gpt/efiesp: FILESYSTEM CLEAN; SKIPPING CHECKS
Growing root partition to fill device
vtbd0 recovering is not needed
gpart: autofill: No space left on device
Mounting local filesystems:.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32
Setting hostname: freebsd.
Setting up harvesting:
PURE_VMGENID,PURE_RDRAND,[CALLOUT],[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy: .
lo0: link state changed to UP
vtnet0: link state changed to UP
Starting Network: lo0 vtnet0.
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu
16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
groups: lo
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP>
metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 00:a0:98:f0:55:05
inet6 fe80::2a0:98ff:fef0:5505%vtnet0 prefixlen 64 scopeid 0x1
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
Starting devd.
Starting dhclient.
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 4
DHCPOFFER from 192.168.241.129
DHCPREQUEST on vtnet0 to 255.255.255.255 port 67
DHCPACK from 192.168.241.129
bound to 192.168.241.132 -- renewal in 17036 seconds.
add host 127.0.0.1: gateway lo0 fib 0: route already in table
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Updating motd:.
Updating /var/run/os-release done.
Clearing /tmp (X related).
Creating and/or trimming log files.
Starting syslogd.
Mounting late filesystems:.
Starting cron.
Starting background file system checks in 60 seconds.
Thu Jun 12 11:24:26 UTC 2025
FreeBSD/amd64 (freebsd) (ttyu0)
login: root
Jun 12 11:24:28 freebsd login[844]: ROOT LOGIN (root) ON ttyu0
FreeBSD 14.3-RELEASE (GENERIC) releng/14.3-n271432-8c9ce319fef7
Welcome to FreeBSD!
Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories: https://www.FreeBSD.org/security/
FreeBSD Handbook: https://www.FreeBSD.org/handbook/
FreeBSD FAQ: https://www.FreeBSD.org/faq/
Questions List: https://www.FreeBSD.org/lists/questions/
FreeBSD Forums: https://forums.FreeBSD.org/
Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with: pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.
Show the version of FreeBSD installed: freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages: man man
FreeBSD directory layout: man hier
To change this login announcement, see motd(5).
root@freebsd:~ # periodic daily
panic: VERIFY3(zfs_btree_find(tree, node, &where) == NULL) failed
(0xfffff8001cc99ba0x == 0x)
cpuid = 1
time = 1749727473
KDB: stack backtrace:
#0 0xffffffff80ba8f1d at kdb_backtrace+0x5d
#1 0xffffffff80b5aa11 at vpanic+0x161
#2 0xffffffff82181c4a at spl_panic+0x3a
#3 0xffffffff821d4aae at zfs_btree_add+0x6e
#4 0xffffffff82315905 at zap_lockdir_impl+0x325
#5 0xffffffff823155a4 at zap_lockdir+0x84
#6 0xffffffff82318b15 at zap_cursor_retrieve+0x175
#7 0xffffffff821a01a3 at zfs_freebsd_readdir+0x383
#8 0xffffffff81112050 at VOP_READDIR_APV+0x20
#9 0xffffffff80c56a19 at kern_getdirentries+0x229
#10 0xffffffff80c56dc9 at sys_getdirentries+0x29
#11 0xffffffff8104e4c7 at amd64_syscall+0x117
#12 0xffffffff8102469b at fast_syscall_common+0xf8
Uptime: 24s
Automatic reboot in 15 seconds - press a key on the console to abort
--> Press a key on the console to reboot,
--> or switch off the system now.