Re: Fun with RDMA and NFS

On Sat, Aug 8, 2026 at 2:53 PM Konstantin Belousov <kib@freebsd.org> wrote:
>
> On Sat, Aug 08, 2026 at 01:15:00PM -0700, Rick Macklem wrote:
> > Hi,
> >
> > As you all no doubt remember, in around 1986, Kirk proposed
> > something like this:
> > #ifdef notdef
> >     /* Not Yet.. */
> >     if (uiop->uio_iov->iov_op != NULL)
> >           (*(uiop->uio_iov->iov_op))(mbufcp, uiocp, xfer);
> >     else
> > #endif
> > The idea was that iov_op would "copy" data without
> > actually doing a memory->memory copying. It never
> > happened, afaik.
> >
> > Well, after only 40years, it looks like NFS might be
> > able to do that.
> >
> > For example, with RDMA a Read will look something like:
> > - NFS VOP_READ() allocates a buffer cache block and
> >   passes the pages (in b_pages[]) to the RPC code.
> >   - The RPC code will map the pages into bus dma space
> >     and make the RPC call (referring to the memory region
> >     for these pages). (What ofed calls FRWR.)
> > On the NFS server...
> > - The server will allocate pages for the read reply data and
> >   map those into bus address space.
> >   It will make a VOP_READ() call to copy the data into these
> >   pages.
> >   The RPC code will get the NIC to copy the data from the pages
> >   to the pages on the NFS client (the buffer cache block in the client
> >   using that FRWR stuff) and then send a small RPC reply noting
> >   the Read has completed.
> > A write would be similar, except the data is copied into the NFS
> > server's pages before the VOP_WRITE().
> >
> > So, at this point, there is still a memory->memory copy done
> > in the exported file system's VOP_READ()/VOP_WRITE().
> >
> > I think that it should be possible to implement a couple of
> > new VOP_xxx() calls to avoid this memory->memory copy.
> > VOP_PAGEIO() - Would return an array of pages with the
> >     read data in them or where the data can be written into
> >     them (it could just return a "struct iovec iov[]"
> >     with the vm_page_t for each page in the iov_base entries).
> >     Presumably the pages would be buffer cache or ARC or ??
> >     for the server file system.
> > VOP_DONE_PAGEIO() - Would be called once the transfer
> >     is done to release the pages.
> >
> > Does this sound feasible?
> Yes, I think it is reasonable.
>
> Perhaps the following signature would be ok:
> int VOP_PAGEIO(struct vnode *vp, foff_t offset, size_t nbytes,
>     enum uio_rw dir, vm_page_t ma[], int max_ma_len, int *ma_len);
> The VOP would return the ma array filled with the busied pages, which
> provide the file content from trunc_page(offset) to round_page(offset+nbytes).
>
> The twist there is that ma_len returns the number of pages that are
> actually filled into the ma array. For instance, for UFS, it is not
> feasible to fill more than a single buffer in time.
>
> From the first look, this is quite easy for UFS and tmpfs.
> If you want I can write the prototype.
Sounds fine to me. No rush, it'll be a while before I get through
initial testing and, since that is the server side, I won't be looking
at it until after that.

>
> >
> > On the NFS client end, it should also be possible for O_DIRECT
> > to have the process's buffer mapped in, so that there is no
> > need for a buffer cache block. (This is more overhead than
> > the FRWR that can be used for kernel pages, but still might
> > be worth the effort.)
> But then we loose the cache coherency.
> Otherwise yes, vm_fault_quick_hold_pages() would provide the ma array for
> the user io buffer.  The easiest example is probably kern/kern_physio.c.
I'm not worried about O_DIRECT, at least for now. It currently works
through the buffer cache by default, and that should work the same as
without O_DIRECT (except when the read and write RPCs are done).
(To be honest, my current prototype code just refuses to allow
RDMA mounts when newnfs_directio_enable (the sysctl variable) when
it is set. It is 0 by default.

Thanks, rick

>
> >
> > So, after years of procrastinating on this, I finally got prodded
> > to do this, thanks to Vinicius's NFS-over-RDMA server work.
> >
> > And, thanks to the Netperf folk, I now have a way of testing
> > NFS-over-RDMA code. (mike@sentex.net, bz@, pho@
> > and others)
> >
> > Things that I thought others might be able to provide help with are:
> > - Creation of the above VOP_xxx() calls { call them whatever
> >   you like } and implementation of them for UFS, ZFS, etc.
> >
> > - If you have servers with RDMA capable hardware, you could
> >   "ibv_devinfo -v" and email me what it dumps out. (If it doesn't
> >   find any configured IB devices, it might just need a driver loaded.
> >   For example, Mellanox requires mlx5ib to be loaded. Not so sure
> >   about Chelsio or Intel?)
> >   This info is useful to me, since it tells me what capabilities I can
> >   count on from the NIC, such as how many scatter/gather entries
> >   it supports.
> >
> > Vinicius has done the server side of NFS-over-RDMA:
> > https://github.com/viniciusferrao/freebsd-src/pull/1
> > and I am working on client side code.
> > - Testing. Although I haven't talked to Vinicius yet, I hope he can
> >   set up a fork/branch of freebsd-src that has what he thinks others
> >   should use for testing and keep that branch relatively up-to-date
> >   with FreeBSD's main.
> >   Once I get the NFS client code I am now testing working ok, I
> >   plan on committing it to main.
> >   --> Then Vinicius's fork/branch will pick it up and provide a place
> >         where others can download sources for testing.
> >
> > Vinicius's code was created with AI usage, so it will be months
> > before I know if it can be pulled into main. If not, maybe it can
> > become a module in ports or ??
> > (But please, please do not make this email thread an AI discussion;-)
> >
> > Just thought some of you might find this interesting and wanted
> > to thank those that have already provided assistance, rick
> > ps: Please let me know if you get working on the new VOP_xxx()
> >       calls.
>