[Bug 247668] Page fault in zfsctl_snapdir_getattr

Tue Jun 30 20:33:39 UTC 2020

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247668

            Bug ID: 247668
           Summary: Page fault in zfsctl_snapdir_getattr
           Product: Base System
           Version: 12.1-STABLE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Many People
          Priority: ---
         Component: kern
          Assignee: bugs at FreeBSD.org
          Reporter: asomers at FreeBSD.org

On a very heavily loaded server I observed the following kernel-mode page
fault.  The offending process was a "procstat -af", which did VOP_GETATTR on
every open file descriptor on the whole system, including the .zfs/snapshot
directories.  On one of those, it called dsl_dataset_phys, which tried to
dereference a null pointer.  There were also 5 "zfs destroy" processes, and
dozens of "zfs list" and "zfs recv" running concurrently.

I suspect that zfsctl_snapdir_getattr is missing some lock when it checks
dsl_dataset_phys, while trying to calculate the directory's nlink attribute. 
But it's not clear what lock it ought to hold.  It's worth noting that ZoL
doesn't have this problem because it doesn't even try to calculate nlink;
instead it always returns "2".

Sadly, I haven't been able to reproduce the issue on any non-production
machine.  

The server in question is running 12-STABLE at svn r346022.

#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:371
#2  0xffffffff80bbe655 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:451
#3  0xffffffff80bbea96 in vpanic (fmt=<optimized out>, ap=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:880
#4  0xffffffff80bbe8b3 in panic (fmt=<unavailable>)
    at /usr/src/sys/kern/kern_shutdown.c:807
#5  0xffffffff81090310 in trap_fatal (frame=0xfffffe04b95c08a0, eva=24)
    at /usr/src/sys/amd64/amd64/trap.c:925
#6  0xffffffff8109035f in trap_pfault (frame=0xfffffe04b95c08a0,
    usermode=<optimized out>, signo=<optimized out>, ucode=<optimized out>)
    at /usr/src/sys/amd64/amd64/trap.c:743
#7  0xffffffff8108f9b8 in trap (frame=0xfffffe04b95c08a0)
    at /usr/src/sys/amd64/amd64/trap.c:407
#8  <signal handler called>
#9  0xffffffff825f4cbc in dsl_dataset_phys (ds=0xfffff86821e72e10)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h:257
#10 zfsctl_snapdir_getattr (ap=<optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ctldir.c:1133
#11 0xffffffff81211315 in VOP_GETATTR_APV (
    vop=0xffffffff826be060 <zfsctl_ops_snapdir>, a=0xfffffe04b95c0a98)
    at vnode_if.c:733
#12 0xffffffff80c7bd29 in VOP_GETATTR (vp=0x1, vap=<optimized out>,
    cred=0xfffff88e58a45700) at ./vnode_if.h:309
#13 vop_stdvptocnp (ap=<optimized out>) at /usr/src/sys/kern/vfs_default.c:743
#14 0xffffffff8121495b in VOP_VPTOCNP_APV (
    vop=0xffffffff81b281b8 <default_vnodeops>, a=0xfffffe04b95c0d90)
    at vnode_if.c:3718
#15 0xffffffff80c78304 in VOP_VPTOCNP (vp=0x0, vpp=<optimized out>,
    cred=0xfffff88e58a45700, buf=0xfffff86ed5d7d400 "",
    buflen=0xfffffe04b95c0e34) at ./vnode_if.h:1599
#16 vn_vptocnp (vp=0xfffffe04b95c0e28, cred=<optimized out>,
    buf=<optimized out>, buflen=<optimized out>)
    at /usr/src/sys/kern/vfs_cache.c:2296
#17 0xffffffff80c77db7 in vn_fullpath1 (td=0xfffff865848d7000,
    vp=0xfffff80e4a8a53c0, rdir=0xfffff860440f0b40, buf=0xfffff86ed5d7d400 "",
    retbuf=0xfffffe04b95c0fa8, buflen=1023)
    at /usr/src/sys/kern/vfs_cache.c:2392
#18 0xffffffff80c780f8 in vn_fullpath (td=0xfffff865848d7000,
    vn=0xfffff80e4a8a53c0, retbuf=0xfffff865848d75a0,
    freebuf=0xfffffe04b95c0fb0) at /usr/src/sys/kern/vfs_cache.c:2221
#19 0xffffffff80ca0635 in vn_fill_kinfo_vnode (vp=0xfffff80e4a8a53c0,
    kif=0xfffff831bcf5e818) at /usr/src/sys/kern/vfs_vnops.c:2352
#20 0xffffffff80c9d3f6 in vn_fill_kinfo (fp=<optimized out>,
    kif=0xfffff831bcf5e818, fdp=<optimized out>)
    at /usr/src/sys/kern/vfs_vnops.c:2318
#21 0xffffffff80b6ca25 in fo_fill_kinfo (fp=<optimized out>,
    kif=<optimized out>, fdp=<optimized out>) at /usr/src/sys/sys/file.h:407
#22 export_file_to_kinfo (fp=<optimized out>, fd=<optimized out>,
    rightsp=<optimized out>, kif=<optimized out>, fdp=0xfffff86618252450,
    flags=1) at /usr/src/sys/kern/kern_descrip.c:3494
#23 export_file_to_sb (fp=0xfffff8210a788460, fd=4, rightsp=<optimized out>,
    efbuf=<optimized out>) at /usr/src/sys/kern/kern_descrip.c:3560
#24 kern_proc_filedesc_out (p=<optimized out>, sb=<optimized out>,
    maxlen=<optimized out>, flags=-1124734960)
    at /usr/src/sys/kern/kern_descrip.c:3667
#25 0xffffffff80b6dbbd in sysctl_kern_proc_filedesc (oidp=<optimized out>,
    arg1=0xfffffe04b95c12bc, arg2=<optimized out>, req=<optimized out>)
    at /usr/src/sys/kern/kern_descrip.c:3701
#26 0xffffffff80bcd639 in sysctl_root_handler_locked (
    oid=0xffffffff81b0a760 <sysctl___kern_proc_filedesc>,
    arg1=0xfffffe04b95c12bc, arg2=1, req=0xfffffe04b95c11f0,
    tracker=0xfffffe04b95c1168) at /usr/src/sys/kern/kern_sysctl.c:166
#27 0xffffffff80bcccf9 in sysctl_root (oidp=<optimized out>,
    arg1=0xfffffe04b95c12bc, arg2=1, req=0xfffffe04b95c11f0)
    at /usr/src/sys/kern/kern_sysctl.c:2062
#28 0xffffffff80bcd368 in userland_sysctl (td=0xfffff865848d7000,
    name=0xfffffe04b95c12b0, namelen=4, old=<optimized out>,
    oldlenp=<optimized out>, inkernel=<optimized out>, new=0x0, newlen=0,
    retval=0xfffffe04b95c1318, flags=0) at /usr/src/sys/kern/kern_sysctl.c:2157
#29 0xffffffff80bcd1af in sys___sysctl (td=0xfffff865848d7000,
    uap=0xfffff865848d73c0) at /usr/src/sys/kern/kern_sysctl.c:2092
#30 0xffffffff81090e87 in syscallenter (td=0xfffff865848d7000)
    at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135
#31 amd64_syscall (td=0xfffff865848d7000, traced=0)
    at /usr/src/sys/amd64/amd64/trap.c:1168
#32 <signal handler called>
#33 0x000000080045789a in ?? ()

-- 
You are receiving this mail because:
You are the assignee for the bug.