Re: RFC: Kernel virtiofs driver
- In reply to: Oleg Sidorkin : "Re: RFC: Kernel virtiofs driver"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Thu, 24 Sep 2026 13:43:26 UTC
чт, 24 сент. 2026 г. в 16:41, Oleg Sidorkin <osidorkin@gmail.com>:
>
> пт, 11 сент. 2026 г. в 09:58, Emil Tsalapatis <emil@etsalapatis.com>:
> >
> > Hi Oleg,
>
> Hi
>
> >
> > Sorry for the long overdue reply, I missed this email. Yes, the three
> > diffs are D46295/D46296/D55047. I will also take a look, but can you
> > see if you can reproduce this with current HEAD? This looks like a bug
> > in the FUSE cached attributes locking that was overhauled this year.
> > so the logic for this may have been changed in the meantime.
>
> I rebuild recent current (commit
> 14c7492637ec0644b5bb88da114e8cc76b9bf629) and tried it on 4 Core VM:
>
> root@freebsd:/usr/src # mount
> /dev/gpt/rootfs on / (ufs, local, noatime, soft-updates, journaled soft-updates)
> devfs on /dev (devfs)
> /dev/gpt/efiboot0 on /boot/efi (msdosfs, local)
> share on /usr/obj (virtiofs)
>
> root@freebsd:/usr/src # make -j2 buildworld buildkernel packages
>
> Failed quite soon:
> ===> usr.bin/dtc (obj,all,install)
> make[3]: /usr/obj/usr/src/amd64.amd64/tmp/obj-tools/usr.bin/dtc/.depend:2:
> Zero byte read from file
> in make[3] in directory "/usr/src/usr.bin/dtc"
>
> make[2]: stopped making "bootstrap-tools" in /usr/src
> --- _bootstrap-tools-lib/clang/libllvmminimal ---
>
> make[2]: stopped making "bootstrap-tools" in /usr/src
> 2.81 real 2.08 user 0.92 sys
>
>
> root@freebsd:/usr/src # make buildworld buildkernel packages
> (single threaded) still works after several minutes. It will take
> about several hours to rebuild everything.
UPD: It failed :(
===> usr.bin/clang/llvm-min-tblgen (obj,all,install)
[Creating objdir
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/usr.bin/clang/llvm-min-tblgen...]
[Creating nested objdir
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/usr.bin/clang/llvm-min-tblgen/Basic...]
echo llvm-min-tblgen.full: /usr/lib/libc.a
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/clang/libllvmminimal/libllvmminimal.a
/usr/lib/libexecinfo.a /usr/lib/libelf.a /usr/lib/libtinfow.a
/usr/lib/libpthread.a
/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/lib/libegacy.a >> .depend
echo llvm-min-tblgen.full: /usr/lib/libc++.a >> .depend
echo: write error on stdout
*** Error code 2
Stop.
make[3]: stopped making "all" in /usr/src/usr.bin/clang/llvm-min-tblgen
*** Error code 1
Stop.
make[2]: stopped making "bootstrap-tools" in /usr/src
272.65 real 243.21 user 13.45 sys
*** Error code 1
Stop.
make[1]: stopped making "buildworld" in /usr/src
*** Error code 1
Stop.
make: stopped making "buildworld buildkernel packages" in /usr/src
>
> So it definitely works better than before, however strange things are
> still happening there.
>
> Great thanks for your work
>
> >
> > Thanks,
> > Emil
> >
> > On Tue, May 5, 2026 at 9:04 AM Oleg Sidorkin <osidorkin@gmail.com> wrote:
> > >
> > > пт, 13 февр. 2026 г. в 05:01, Emil Tsalapatis <emil@etsalapatis.com>:
> > > >
> > > > On Thu, Feb 12, 2026 at 9:01 AM Oleg Sidorkin <osidorkin@gmail.com> wrote:
> > > >>
> > > >> Hello.
> > > >>
> > > >> My favorite test to run in bhyve guest:
> > > >>
> > > >> root@:/usr/src # uname -a
> > > >> FreeBSD 16.0-CURRENT FreeBSD 16.0-CURRENT #1
> > > >> main-n283712-16c902f05853: Sat Feb 7 02:10:47 MSK 2026
> > > >> olsi@:/usr/obj/usr/src/amd64.amd64/sys/QUADKERNEL amd64
> > > >> root@:/usr/src # mount
> > > >> /dev/vtbd0p2 on / (ufs, local, soft-updates, journaled soft-updates)
> > > >> devfs on /dev (devfs)
> > > >> obj on /usr/obj (p9fs, local)
> > > >> 192.168.2.1:/vms/freebsd-current/usr/home on /usr/home (nfs)
> > > >> 192.168.2.1:/vms/freebsd-current/usr/src on /usr/src (nfs)
> > > >> 192.168.2.1:/vms/freebsd-current/usr/lib/debug on /usr/lib/debug (nfs)
> > > >> 192.168.2.1:/usr/ports/distfiles on /usr/ports/distfiles (nfs)
> > > >> root@:/usr/src # make -j4 buildworld buildkernel
> > > >>
> > > >> Panics guest system in a few minutes (everything is ok when /obj is
> > > >> mounted over nfs):
> > > >>
> > > >> db> bt
> > > >> Tracing pid 2248 tid 100165 td 0xfffff80100eb8780
> > > >> kdb_enter() at kdb_enter+0x33/frame 0xfffffe006833e560
> > > >> panic() at panic+0x43/frame 0xfffffe006833e5c0
> > > >> freevnode() at freevnode+0x2d5/frame 0xfffffe006833e620
> > > >> vput_final() at vput_final+0x96/frame 0xfffffe006833e670
> > > >> vfs_hash_insert() at vfs_hash_insert+0x226/frame 0xfffffe006833e6c0
> > > >> p9fs_vget_common() at p9fs_vget_common+0x39b/frame 0xfffffe006833e770
> > > >> p9fs_lookup() at p9fs_lookup+0x4ad/frame 0xfffffe006833e8c0
> > > >> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x57/frame 0xfffffe006833e8f0
> > > >> vfs_lookup() at vfs_lookup+0x5aa/frame 0xfffffe006833e980
> > > >> namei() at namei+0x35d/frame 0xfffffe006833e9e0
> > > >> kern_execve() at kern_execve+0x2d1/frame 0xfffffe006833ed80
> > > >> sys_execve() at sys_execve+0x54/frame 0xfffffe006833ee00
> > > >> amd64_syscall() at amd64_syscall+0x169/frame 0xfffffe006833ef30
> > > >> fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe006833ef30
> > > >> --- syscall (59, FreeBSD ELF64, execve), rip = 0x2cc73682f46a, rsp =
> > > >> 0x2cc731eb2028, rbp = 0x2cc731eb2170 ---
> > > >> db>
> > > >>
> > > >> I'll be really happy to test virtiofs and I hope it will do better.
> > > >>
> > > >
> > > > If you hit any bugs, let me know! I'd be happy to take a look. Please include some additional patches apart from the two diffs (https://github.com/etsal/freebsd-src/tree/virtiofs), that handle some issues virtiofs was triggering in the FUSE layer (D55047, D55046).
> > > >
> > > > Also, if you could try running with a debugging kernel I'd really appreciate it - it will help us better interpret any issues. For example, with a config like this:
> > > >
> > > > include GENERIC
> > > >
> > > > ident DEBUG
> > > >
> > > > options KDB # Enable kernel debugger support.
> > > > options KDB_TRACE # Print a stack trace for a panic.
> > > > options DDB
> > > > options DDB_NUMSYM
> > > > options GDB
> > > >
> > > > options INVARIANTS
> > > > options INVARIANT_SUPPORT
> > > > options DIAGNOSTIC
> > > >
> > > > # KTR
> > > > options KTR
> > > > options ALQ
> > > > options KTR_ALQ
> > > >
> > > > # KASAN
> > > > options KASAN
> > > >
> > > >
> > > > Thanks,
> > > > Emil
> > > >
> > > >
> > > >
> > > >>
> > > >> Thanks
> > > >>
> > > >> вт, 10 февр. 2026 г. в 09:57, Mario Marietto <marietto2008@gmail.com>:
> > > >> >
> > > >> > Hello Emil,
> > > >> >
> > > >> > Inside a FreeBSD guest OS (15.0-RELEASE) I do :
> > > >> >
> > > >> > kldload virtio_p9fs
> > > >> >
> > > >> > kldload p9fs_load
> > > >> >
> > > >> > mount -t p9fs sharename /mnt/host
> > > >> >
> > > >> > This works for me,I can share files between FreeBSD 15.0 guest and FreeBSD 14.3 host os. So,what's missing in this case and which features you added ?
> > > >> >
> > > >> > Thanks.
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Tue, Feb 10, 2026 at 4:05 AM Emil Tsalapatis <emil@etsalapatis.com> wrote:
> > > >> >>
> > > >> >> Hi everyone,
> > > >> >>
> > > >> >> I recently finished the virtiofs driver and it is now ready for review. The device allows for sharing directories between a FreeBSD guest and a host.
> > > >> >>
> > > >> >> The driver really is two components:
> > > >> >>
> > > >> >> 1) The virtio device that sends FUSE tickets to and from the host: D46295
> > > >> >> 2) The file system that gets mounted in the guest: D46296.
> > > >> >>
> > > >> >> To test it you need a couple additional fixes/workarounds for FUSE-related issues. You can grab a working tree here or apply diffs D55047 and D55046. D55046 is a workaround, but still prevents an assertion failure related to FUSE caching until the underlying issue is properly fixed on HEAD.
> > > >> >>
> > > >> >> To use it, make sure you are creating virtiofs device on the host then from the FreeBSD guest run
> > > >> >>
> > > >> >> mount -t virtiofs <tag> <mountpoint>
> > > >> >>
> > > >> >> where <tag> is the name tag you gave to the virtiofs device in the host VMM.
> > > >> >>
> > > >> >> Reviews and testing welcome!
> > > >> >>
> > > >> >> Thanks,
> > > >> >> Emil
> > > >> >>
> > > >> >>
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Mario.
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Oleg Sidorkin
> > >
> > > Hi, I finally was able to set up an appropriate VM to test it.
> > > I applied 3 of 4 patches (all except
> > > https://reviews.freebsd.org/D55046 that seems to be already applied in
> > > a different way).
> > >
> > > It worked fine for simple file operations like creating and editing
> > > text files, but when I tried to use virtiofs mount as /usr/obj, it
> > > panicked almost instantly with the following backtrace
> > >
> > > #11 0xffffffff80bdc0a3 in panic (
> > > fmt=0xffffffff81da22a0 <cnputs_mtx> "\017\344\"\201\377\377\377\377")
> > > at /usr/src/sys/kern/kern_shutdown.c:887
> > > #12 0xffffffff82e3a7c9 in fuse_vnop_close (ap=<optimized out>)
> > > at /usr/src/sys/fs/fuse/fuse_vnops.c:910
> > > #13 0xffffffff81211c22 in VOP_CLOSE_APV (vop=0xffffffff82e467a8 <fuse_vnops>,
> > > a=a@entry=0xfffffe0068343ce8) at vnode_if.c:469
> > > rc = <optimized out>
> > > #14 0xffffffff80b82411 in VOP_CLOSE (vp=<optimized out>, fflag=1, cred=0x12,
> > > #15 do_execve (td=0xfffff80100bc3000, args=0xfffffe0068343d98,
> > > mac_p=<optimized out>, oldvmspace=<optimized out>)
> > > at /usr/src/sys/kern/kern_exec.c:978
> > >
> > > (I can paste the entire stracktrace if needed).
> > >
> > > It looks like this assert:
> > > https://github.com/freebsd/freebsd-src/blob/dc00f118405e8638ceb13b288e14164a8a9ba669/sys/fs/fuse/fuse_vnops.c#L910
> > > added to recent -CURRENT, however I couldn't find where the lock
> > > should be taken.
> > >
> > >
> > > Thanks
> > >
> > > Oleg Sidorkin
>
>
>
> --
> Oleg Sidorkin
--
Oleg Sidorkin