[Bug 266014] panic: on long running find (zfs issue)

From: <bugzilla-noreply_at_freebsd.org>
Date: Sun, 04 Sep 2022 12:34:54 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=266014

--- Comment #4 from Duncan <dpy@pobox.com> ---
I have compiled my system from source and caused a panic (run periodic daily
(daily_clean_disks_enabled will cause it) ). I turned off all the VMs and jails
before causing the crash (remove extra complexity and reduce the size of the
core.txt file). I also added a larger dump disk and now have a complete viable
dump.  I have attached the complete core.txt.0 file and the following is the
starting portion:

triple0.internal dumped core - see /var/crash/vmcore.0

Sun Sep  4 20:09:55 AEST 2022

FreeBSD triple0.internal 13.1-RELEASE-p2 FreeBSD 13.1-RELEASE-p2
releng/13.1-n250158-752f813d6cc GENERIC  amd64

panic: VERIFY3(0 == sa_handle_get_from_db(zfsvfs->z_os, db, zp, SA_HDL_SHARED,
&zp->z_sa_hdl)) failed (0 == 5)

GNU gdb (GDB) 12.1 [GDB v12.1 for FreeBSD]
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd13.1".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:
panic: VERIFY3(0 == sa_handle_get_from_db(zfsvfs->z_os, db, zp, SA_HDL_SHARED,
&zp->z_sa_hdl)) failed (0 == 5)

cpuid = 8
time = 1662284816
KDB: stack backtrace:
#0 0xffffffff80c694a5 at kdb_backtrace+0x65
#1 0xffffffff80c1bb5f at vpanic+0x17f
#2 0xffffffff8500bf4a at spl_panic+0x3a
#3 0xffffffff85032c12 at zfs_znode_alloc+0x522
#4 0xffffffff85033c35 at zfs_zget+0x3b5
#5 0xffffffff8501e84b at zfs_dirent_lookup+0x16b
#6 0xffffffff8501e91a at zfs_dirlook+0x7a
#7 0xffffffff85030a10 at zfs_lookup+0x3d0
#8 0xffffffff8502c11b at zfs_freebsd_cachedlookup+0x6b
#9 0xffffffff80cdc4ed at vfs_cache_lookup+0xad
#10 0xffffffff80ce0cf0 at VOP_LOOKUP+0x30
#11 0xffffffff80ce0723 at cache_fplookup_noentry+0x1a3
#12 0xffffffff80cddfa6 at cache_fplookup+0x366
#13 0xffffffff80ce8cfa at namei+0x12a
#14 0xffffffff80d06993 at kern_statat+0xf3
#15 0xffffffff80d0708f at sys_fstatat+0x2f
#16 0xffffffff810b06ec at amd64_syscall+0x10c
#17 0xffffffff81087e8b at fast_syscall_common+0xf8
Uptime: 56m13s
(ada0:ahcich1:0:0:0): spin-down
(ada1:ahcich2:0:0:0): spin-down
(ada2:ahcich3:0:0:0): spin-down
(ada3:ahcich4:0:0:0): spin-down
Dumping 31966 out of 130858
MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct
pcpu,
(kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:399
#2  0xffffffff80c1b75c in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:487
#3  0xffffffff80c1bbce in vpanic (
    fmt=0xffffffff8527c012 "VERIFY3(0 == sa_handle_get_from_db(zfsvfs->z_os,
db, zp, SA_HDL_SHARED, &zp->z_sa_hdl)) failed (0 == %lld)\n",
    ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:920
#4  0xffffffff8500bf4a in spl_panic (file=<optimized out>,
    func=<optimized out>, line=<unavailable>, fmt=<unavailable>)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/spl/spl_misc.c:107
#5  0xffffffff85032c12 in zfs_znode_sa_init (zfsvfs=0xfffff8027a9c5000,
    zp=0xfffff807415f9ce8, db=0xfffff80f7c49acb8, obj_type=DMU_OT_SA,
    sa_hdl=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_znode.c:364
#6  zfs_znode_alloc (zfsvfs=zfsvfs@entry=0xfffff8027a9c5000,
    db=0xfffff80f7c49acb8, blksz=3584, obj_type=DMU_OT_SA, hdl=hdl@entry=0x0)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_znode.c:466
#7  0xffffffff85033c35 in zfs_zget (zfsvfs=<optimized out>,
    zfsvfs@entry=0xfffff8027a9c5000, obj_num=<optimized out>,
    zpp=zpp@entry=0xfffffe03109916f0)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_znode.c:1042
#8  0xffffffff8501e84b in zfs_dirent_lookup (
    dzp=dzp@entry=0xfffff80741474588, name=0xfffffe0310991860 "tb_pkmeth.c",
    zpp=zpp@entry=0xfffffe0310991740, flag=flag@entry=2)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_dir.c:191
#9  0xffffffff8501e91a in zfs_dirlook (dzp=dzp@entry=0xfffff80741474588,
    name=<unavailable>, name@entry=0xfffffe0310991860 "tb_pkmeth.c",
    zpp=zpp@entry=0xfffffe03109917e0)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_dir.c:247
#10 0xffffffff85030a10 in zfs_lookup (dvp=<optimized out>,
    nm=nm@entry=0xfffffe0310991860 "tb_pkmeth.c", vpp=<optimized out>,
    cnp=cnp@entry=0xfffffe0310991c58, nameiop=0, cr=<optimized out>, flags=0,
    cached=1)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:929
#11 0xffffffff8502c11b in zfs_freebsd_lookup (ap=0xfffffe0310991990,
    ap@entry=<error reading variable: value is not available>, cached=1)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:4593
#12 zfs_freebsd_cachedlookup (ap=0xfffffe0310991990,
    ap@entry=<error reading variable: value is not available>)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:4601
#13 0xffffffff80cdc4ed in VOP_CACHEDLOOKUP (dvp=0xfffff80730da45b8,
    vpp=0xfffffe0310991a10, cnp=0xfffffe0310991c58) at ./vnode_if.h:99
#14 vfs_cache_lookup (ap=<unavailable>,
    ap@entry=<error reading variable: value is not available>)
    at /usr/src/sys/kern/vfs_cache.c:3069
#15 0xffffffff80ce0cf0 in VOP_LOOKUP (dvp=dvp@entry=0xfffff80730da45b8,
    vpp=<unavailable>, vpp@entry=0xfffffe0310991a10, cnp=<unavailable>,
    cnp@entry=0xfffffe0310991c58) at ./vnode_if.h:65
#16 0xffffffff80ce0723 in cache_fplookup_noentry (
    fpl=fpl@entry=0xfffffe0310991a88) at /usr/src/sys/kern/vfs_cache.c:4928
#17 0xffffffff80cddfa6 in cache_fplookup_next (fpl=0xfffffe0310991a88)
    at /usr/src/sys/kern/vfs_cache.c:5284
#18 cache_fplookup_impl (dvp=<optimized out>, fpl=0xfffffe0310991a88)
    at /usr/src/sys/kern/vfs_cache.c:5932
#19 cache_fplookup (ndp=ndp@entry=0xfffffe0310991bd8,
    status=status@entry=0xfffffe0310991b84,
    pwdp=pwdp@entry=0xfffffe0310991b88) at /usr/src/sys/kern/vfs_cache.c:6104
#20 0xffffffff80ce8cfa in namei (ndp=ndp@entry=0xfffffe0310991bd8)
    at /usr/src/sys/kern/vfs_lookup.c:570
#21 0xffffffff80d06993 in kern_statat (td=0xfffffe030c6b5000,
    flag=<optimized out>, fd=-100, path=<unavailable>, pathseg=<unavailable>,
    pathseg@entry=UIO_USERSPACE, sbp=sbp@entry=0xfffffe0310991d18, hook=0x0)
    at /usr/src/sys/kern/vfs_syscalls.c:2441
#22 0xffffffff80d0708f in sys_fstatat (td=<unavailable>,
    uap=0xfffffe030c6b53e8) at /usr/src/sys/kern/vfs_syscalls.c:2418
#23 0xffffffff810b06ec in syscallenter (td=0xfffffe030c6b5000)
    at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:189
#24 amd64_syscall (td=0xfffffe030c6b5000, traced=0)
    at /usr/src/sys/amd64/amd64/trap.c:1185
#25 <signal handler called>
#26 0x00000008011ad39a in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffffffd588
(kgdb)

------------------------------------------------------------------------

-- 
You are receiving this mail because:
You are the assignee for the bug.