Re: Implementing VOP_READPLUS() in FreeBSD 15?

On Fri, 7 Nov 2025 at 03:02, Rick Macklem <rick.macklem@gmail.com> wrote:
>
> On Thu, Nov 6, 2025 at 11:40 AM Aurélien Couderc
> <aurelien.couderc2002@gmail.com> wrote:
> >
> > This is a followup to a discussion with the nfs-ganesha developers.
> >
> > Could FreeBSD implement a VOP_READPLUS() in FreeBSD 15, please?
> >
> > Citing Lionel Cons/CERN:
> > > But the point is to optimise the read(). First, you have less traffic over the wire (which is a
> > > thing if your reads are in the gigabyte range for large VMs), and it tells the VM host that it
> > > can just map all those MMU pages representing the hole to the "default zero page", which
> > > in turn saves lots of space in the L3 and L2 caches ----> THIS DOES WONDERS to VM
> > > performance.
> > >
> > > Example:
> > > The performance benefit here comes from the fast that instead of mapping a 1TB hole
> > > (1099511627776 bytes) to individual 524288 2M pages (x86 2M hugepage size), and then
> > > potentially reading from them, you just have ONE 2M page in the cache, and all reads come
> > > from that.
> > >
> > > READ_PLUS is THE game changer for that kind of application, especially in our case (HPC
> > > simulations).
> Why doesn't the application use lseek(SEEK_DATA/SEEK_HOLE) and only read(2) the
> data segments?

Because these are extra syscalls with extra latency, and this is not
applicable to Infiniband/RDMA.

But let's move back in history first:
The filesystem op READ_PLUS (and WRITE_PLUS) comes from Sun
Microsystems. Originally conceived by Eric Blake and Roland Mainz as
part of the "new VM system", and was slated to be integrated in
Solaris 12.
But when ORACLE bought SUN, Solaris 12 was canceled - but at least the
new VM system was backported to Solaris 11.3 and 11.4.
About the fate of the new IO system, I do not know.

How would the new IO system look like:
1. READ_PLUS, to issue a read,  instead of getting a straight buffer
filled it returned a set of ranges+data. Each range is one of DATA,
HOLE, ADB (application data block).
2. WRITE_PLUS is READ_PLUS in reverse, you pass a set of either DATA;
HOLE or ADB to the driver, plus flags which sets can be concurrently
written.

For backwards compatibility there is a method to query which IO calls
are supported by a filesystem, and if no bits of READ_PLUS or
WRITE_PLUS are set, then it will be emulated.

>
> This is implemented now in FreeBSD and in several other POSIX-like OSs
> and avoids
> problems like filling the buffer cache with blocks of all zeros or
> returning a lot of blocks
> with all zeros to the application via read(2).

Yeah, but touching each page you read just to scan for zeros is
inefficient, especially if you have 2M large pages and RDMA. This
kills performance.

>
> Right now, I not aware of any read_plus(2) syscall (please correct me
> if I am wrong on this),
> so applications that read(2) sparse files without bothering to do
> lseek(SEEK_DATA/SEEK_HOLE) will get a lot of 0s to process.
>
> To do VOP_READPLUS() is a lot of work. Once the VOP_READPLUS() is defined,
> there needs to be implementations in the various local fs (ZFS, UFS,
> ..). That requires
> work by people who know these areas. I am only minimally conversant with either
> ZFS or UFS and would not want to attempt to do a good VOP_READPLUS()
> implementation for either of them. (Without fs specific
> implementations, there isn't
> much point in doing it, imho.)

Both UFS and ZFS can run via emulation, until there is a native implementation.

Lionel