git: 1ecbc1d8e9d3 - main - cxgbe tom: Don't queue AIO requests on listen sockets.

[Alan Somers]

On Wed, Sep 15, 2021 at 11:32 AM John Baldwin <jhb at freebsd.org> wrote:

> On 9/15/21 8:47 AM, Alan Somers wrote:
> > On Wed, Sep 15, 2021 at 9:21 AM John Baldwin <jhb at freebsd.org> wrote:
> >
> >> On 9/14/21 1:53 PM, Alan Somers wrote:
> >>> On Tue, Sep 14, 2021 at 2:46 PM John Baldwin <jhb at freebsd.org> wrote:
> >>>
> >>>> The branch main has been updated by jhb:
> >>>>
> >>>> URL:
> >>>>
> >>
> https://cgit.FreeBSD.org/src/commit/?id=1ecbc1d8e9d3fbcd8e68fc68f0a32944a12ddb1e
> >>>>
> >>>> commit 1ecbc1d8e9d3fbcd8e68fc68f0a32944a12ddb1e
> >>>> Author:     John Baldwin <jhb at FreeBSD.org>
> >>>> AuthorDate: 2021-09-14 20:46:14 +0000
> >>>> Commit:     John Baldwin <jhb at FreeBSD.org>
> >>>> CommitDate: 2021-09-14 20:46:14 +0000
> >>>>
> >>>>       cxgbe tom: Don't queue AIO requests on listen sockets.
> >>>>
> >>>>       This is similar to the fixes in 141fe2dceeae.  One difference is
> >> that
> >>>>       TOE sockets do not change states (listen vs non-listen) once
> >> created,
> >>>>       so no lock is needed for SOLISTENING().
> >>>>
> >>>>       Sponsored by:   Chelsio Communications
> >>>>
> >>>
> >>> I've always wondered: what's the point to using AIO with sockets?
> Can't
> >>> everything socket-related be done better with non-blocking read/write
> and
> >>> kqueue?
> >>
> >> Zero-copy operation with TOE is why TOE uses AIO.  Zero-copy of user
> >> buffers
> >> can't really work with the non-AIO APIs because the user buffer is free
> to
> >> be reused immediately after write(2) (and on the read side you don't
> know
> >> the buffer in advance to allow the NIC to write directly into the use
> >> buffer).
> >>
> >> In theory we could support zero-copy using mb_ext_pgs for aio_write()
> for
> >> the non-TOE case similar to what sendfile() does.
> >>
> >> --
> >> John Baldwin
> >>
> >
> > Interesting.  Do you know of any common applications that include this
> > optimization?  I've been working on the AIO ecosystem for Rust.  It would
> > be good to ensure that this use case works, especially if zero-copy ever
> > works for non-TOE.
>
> I do not, and I rely on patches I merged upstream to netperf (-a and -A
> flags)
> to test it.  I believe there might be some proprietary bits in some FreeBSD
> downstreams that might make use of this.
>
> --
> John Baldwin
>

Do you mean these -a and -A flags, or am I looking in the wrong place?
       -a sizespec
              Alter the send and receive buffer alignments on the local
              system.  This defaults to 8 bytes.

       -A sizespec
              As -a, but for the remote system.