[Bug 283448] [fusefs] use after free on NFS-exported file fuse systems

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=283448

            Bug ID: 283448
           Summary: [fusefs] use after free on NFS-exported file fuse
                    systems
           Product: Base System
           Version: 15.0-CURRENT
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: asomers@FreeBSD.org

NFS is weird.  It's stateless, so it never calls VOP_OPEN or VOP_CLOSE.  The
fuse protocol, however, usually requires FUSE_OPEN and FUSE_CLOSE commands to
be sent to the server.  So fusefs(4) has some weird hacks in it to make that
work.  But it's possible to trigger a use-after-free bug in those weird hacks. 
On a debug kernel, it will panic.

The problem is that VOP_READ will open a fuse_filehandle itself, if necessary. 
It should only be necessary when the read is coming from the nfs server.  Then,
VOP_READ will close the fuse_filehandle.  But in the meantime another thread
may be borrowing that fuse_filehandle, and the close isn't properly
synchronized.

To reproduce this bug, you must have a fuse file system that is suitable for
export over NFS.  That is, it does not need FUSE_OPEN or FUSE_OPENDIR,
correctly handles FUSE_LOOKUP for ".", and only uses 32-bit (or less)
generation numbers.  Not all fuse file systems satisfy these requirements.  In
particular, anything that links to libfuse2 will not.  In this example, I use
bfffs.

Steps to Reproduce
==================
0. Create a suitable fusefs file system, and mount it to /mnt
1. export that file system over NFS, in /etc/exports, and start rpcbind and
nfsd
2. Mount it on a client:
  sudo mount -t nfs -o nfsv4,minorversion=2 192.168.0.27:/mnt /mnt
3. Install fsx
   pkg install -y fsx
4. Create an fsx.toml file with these contents:
flen = 1048576
nomsyncafterwrite = false
nosizechecks = false
blockmode = false

[opsize]
max = 262144
min = 0
align = 1

[weights]
close_open = 1
read = 10
write = 10
mapread = 10
mapwrite = 10
invalidate = 1
truncate = 1
fsync = 1
fdatasync = 1
posix_fallocate = 0
punch_hole = 1
sendfile = 1
posix_fadvise = 0
copy_file_range = 1

5. Run fsx on the NFS mount.  I had to run it 4 times to trigger the bug
  sudo fsx -N 1000 -f /tmp/bfffs-on-nfs.toml /mnt/fsx.bin

Stack Trace
===========
panic: page fault
cpuid = 12
time = 1734714504
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0150adf350
vpanic() at vpanic+0x136/frame 0xfffffe0150adf480
panic() at panic+0x43/frame 0xfffffe0150adf4e0
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe0150adf540
trap_pfault() at trap_pfault+0xa0/frame 0xfffffe0150adf5b0
calltrap() at calltrap+0x8/frame 0xfffffe0150adf5b0
--- trap 0xc, rip = 0xffffffff82813b3e, rsp = 0xfffffe0150adf680, rbp =
0xfffffe0150adf690 ---
fuse_ticket_drop() at fuse_ticket_drop+0xe/frame 0xfffffe0150adf690
fuse_internal_fsync() at fuse_internal_fsync+0xf8/frame 0xfffffe0150adf730
nfsvno_fsync() at nfsvno_fsync+0x5f/frame 0xfffffe0150adf7a0
nfsrvd_commit() at nfsrvd_commit+0xf0/frame 0xfffffe0150adf9a0
nfsrvd_dorpc() at nfsrvd_dorpc+0x167e/frame 0xfffffe0150adfbb0
nfssvc_program() at nfssvc_program+0x808/frame 0xfffffe0150adfdb0
svc_run_internal() at svc_run_internal+0xa09/frame 0xfffffe0150adfee0
svc_thread_start() at svc_thread_start+0xb/frame 0xfffffe0150adfef0
fork_exit() at fork_exit+0x82/frame 0xfffffe0150adff30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0150adff30
--- trap 0xc, rip = 0x107988c0c1fa, rsp = 0x107986884028, rbp = 0x1079868842c0
---
KDB: enter: panic

Analysis
========
kgdb shows that within fdisp_destroy, ftick->tick was NULL.  That indicates a
double-free.   Most likely, one thread was running fuse_internal_fsync while
another fuse_vnop_read.  The first thread accessed a fuse_filehandle structure
while the second was freeing it.

-- 
You are receiving this mail because:
You are the assignee for the bug.