Re: Implementing VOP_READPLUS() in FreeBSD 15?

On Wed, Nov 12, 2025 at 3:54 AM Lionel Cons <lionelcons1972@gmail.com> wrote:
>
> On Fri, 7 Nov 2025 at 03:02, Rick Macklem <rick.macklem@gmail.com> wrote:
> >
> > On Thu, Nov 6, 2025 at 11:40 AM Aurélien Couderc
> > <aurelien.couderc2002@gmail.com> wrote:
> > >
> > > This is a followup to a discussion with the nfs-ganesha developers.
> > >
> > > Could FreeBSD implement a VOP_READPLUS() in FreeBSD 15, please?
> > >
> > > Citing Lionel Cons/CERN:
> > > > But the point is to optimise the read(). First, you have less traffic over the wire (which is a
> > > > thing if your reads are in the gigabyte range for large VMs), and it tells the VM host that it
> > > > can just map all those MMU pages representing the hole to the "default zero page", which
> > > > in turn saves lots of space in the L3 and L2 caches ----> THIS DOES WONDERS to VM
> > > > performance.
> > > >
> > > > Example:
> > > > The performance benefit here comes from the fast that instead of mapping a 1TB hole
> > > > (1099511627776 bytes) to individual 524288 2M pages (x86 2M hugepage size), and then
> > > > potentially reading from them, you just have ONE 2M page in the cache, and all reads come
> > > > from that.
> > > >
> > > > READ_PLUS is THE game changer for that kind of application, especially in our case (HPC
> > > > simulations).
> > Why doesn't the application use lseek(SEEK_DATA/SEEK_HOLE) and only read(2) the
> > data segments?
>
> Because these are extra syscalls with extra latency, and this is not
> applicable to Infiniband/RDMA.
Note that FreeBSD's NFS does not currently handle RDMA. Since there
has been no way for me to do testing (FreeBSD does not have the rxe or
siw drivers that Linux does), I haven't worked on it until literally days ago.

I am now making an attempt to get that working over the next year, but
I won't make any guarantees w.r.t. getting it working.
(This is my "for fun" project for the next year.)

>
> But let's move back in history first:
> The filesystem op READ_PLUS (and WRITE_PLUS) comes from Sun
> Microsystems. Originally conceived by Eric Blake and Roland Mainz as
> part of the "new VM system", and was slated to be integrated in
> Solaris 12.
> But when ORACLE bought SUN, Solaris 12 was canceled - but at least the
> new VM system was backported to Solaris 11.3 and 11.4.
> About the fate of the new IO system, I do not know.
>
> How would the new IO system look like:
> 1. READ_PLUS, to issue a read,  instead of getting a straight buffer
> filled it returned a set of ranges+data. Each range is one of DATA,
> HOLE, ADB (application data block).
> 2. WRITE_PLUS is READ_PLUS in reverse, you pass a set of either DATA;
> HOLE or ADB to the driver, plus flags which sets can be concurrently
> written.
>
> For backwards compatibility there is a method to query which IO calls
> are supported by a filesystem, and if no bits of READ_PLUS or
> WRITE_PLUS are set, then it will be emulated.
>
> >
> > This is implemented now in FreeBSD and in several other POSIX-like OSs
> > and avoids
> > problems like filling the buffer cache with blocks of all zeros or
> > returning a lot of blocks
> > with all zeros to the application via read(2).
>
> Yeah, but touching each page you read just to scan for zeros is
> inefficient, especially if you have 2M large pages and RDMA. This
> kills performance.
>
> >
> > Right now, I not aware of any read_plus(2) syscall (please correct me
> > if I am wrong on this),
> > so applications that read(2) sparse files without bothering to do
> > lseek(SEEK_DATA/SEEK_HOLE) will get a lot of 0s to process.
> >
> > To do VOP_READPLUS() is a lot of work. Once the VOP_READPLUS() is defined,
> > there needs to be implementations in the various local fs (ZFS, UFS,
> > ..). That requires
> > work by people who know these areas. I am only minimally conversant with either
> > ZFS or UFS and would not want to attempt to do a good VOP_READPLUS()
> > implementation for either of them. (Without fs specific
> > implementations, there isn't
> > much point in doing it, imho.)
>
> Both UFS and ZFS can run via emulation, until there is a native implementation.
A default VOP_READPLUS() could be implemented using SEEK_DATA/SEEK_HOLE.
My concern is that this will result in a lot of overhead in the
server, since SEEK_DATA/SEEK_HOLE
can be expensive operations.
- For ZFS, they are a killer for recently written files unless the sysctl
  called vfs.zfs..dsync, (sorry I don't have a system handy to look
  up an exact name) is set 0. When set to 0, holes in recently written
  files will be missed, but for VOP_READPLUS() that is only a performance
  issue.
- For UFS, it is non-linear, since large files involve reading/searching more
  indirect and doubly indirect blocks (if I recall my stale UFS correctly).
  Hopefully they all stay in the buffer cache once first read.
Part of the problem is there is no way to know if a file is sparse without
doing the SEEK_DATA/SEEK_HOLE and the worst case is probably
a large file without any holes.
-> VOP_READPLUS (and presumably NFSv4.2's READ_PLUS) will
    end up returning 1Mbyte of data. Then the next call will do the same,
    and so on. But unlike the normal NFS Read, every RPC will do a
    SEEK_DATA/SEEK_HOLE to look for the hole that doesn't exist.

The result of all this is that the emulation isn't going to be very
satisfactory and, if no one is working on the actual VOP_READPLUS()
implementations for these file systems, it will be all there is and
you're stuck with it.
(In Linux, I believe there was an implementation of NFSv4.2 READ_PLUS
that just did Read and didn't even look for holes, but that might have
changed now and I might not be correct w.r.t. this.)

A final reminder. I am one retired guy who does this as a hobby.
If you really want this to happen, finding someone (volunteer or
paid) that can do the VOP_READPLUS() for UFS and ZFS and
contribute good patches that do it would pave the way for NFS
support.

rick

>
> Lionel
>