git: 1ecbc1d8e9d3 - main - cxgbe tom: Don't queue AIO requests on listen sockets.

[Navdeep Parhar]

On 9/15/21 11:05 AM, Alan Somers wrote:
> On Wed, Sep 15, 2021 at 11:32 AM John Baldwin <jhb at freebsd.org 
> <mailto:jhb at freebsd.org>> wrote:
> 
>     On 9/15/21 8:47 AM, Alan Somers wrote:
>      > On Wed, Sep 15, 2021 at 9:21 AM John Baldwin <jhb at freebsd.org
>     <mailto:jhb at freebsd.org>> wrote:
>      >
>      >> On 9/14/21 1:53 PM, Alan Somers wrote:
>      >>> On Tue, Sep 14, 2021 at 2:46 PM John Baldwin <jhb at freebsd.org
>     <mailto:jhb at freebsd.org>> wrote:
>      >>>
>      >>>> The branch main has been updated by jhb:
>      >>>>
>      >>>> URL:
>      >>>>
>      >>
>     https://cgit.FreeBSD.org/src/commit/?id=1ecbc1d8e9d3fbcd8e68fc68f0a32944a12ddb1e
>     <https://cgit.FreeBSD.org/src/commit/?id=1ecbc1d8e9d3fbcd8e68fc68f0a32944a12ddb1e>
>      >>>>
>      >>>> commit 1ecbc1d8e9d3fbcd8e68fc68f0a32944a12ddb1e
>      >>>> Author:     John Baldwin <jhb at FreeBSD.org>
>      >>>> AuthorDate: 2021-09-14 20:46:14 +0000
>      >>>> Commit:     John Baldwin <jhb at FreeBSD.org>
>      >>>> CommitDate: 2021-09-14 20:46:14 +0000
>      >>>>
>      >>>>       cxgbe tom: Don't queue AIO requests on listen sockets.
>      >>>>
>      >>>>       This is similar to the fixes in 141fe2dceeae.  One
>     difference is
>      >> that
>      >>>>       TOE sockets do not change states (listen vs non-listen) once
>      >> created,
>      >>>>       so no lock is needed for SOLISTENING().
>      >>>>
>      >>>>       Sponsored by:   Chelsio Communications
>      >>>>
>      >>>
>      >>> I've always wondered: what's the point to using AIO with
>     sockets?  Can't
>      >>> everything socket-related be done better with non-blocking
>     read/write and
>      >>> kqueue?
>      >>
>      >> Zero-copy operation with TOE is why TOE uses AIO.  Zero-copy of user
>      >> buffers
>      >> can't really work with the non-AIO APIs because the user buffer
>     is free to
>      >> be reused immediately after write(2) (and on the read side you
>     don't know
>      >> the buffer in advance to allow the NIC to write directly into
>     the use
>      >> buffer).
>      >>
>      >> In theory we could support zero-copy using mb_ext_pgs for
>     aio_write() for
>      >> the non-TOE case similar to what sendfile() does.
>      >>
>      >> --
>      >> John Baldwin
>      >>
>      >
>      > Interesting.  Do you know of any common applications that include
>     this
>      > optimization?  I've been working on the AIO ecosystem for Rust. 
>     It would
>      > be good to ensure that this use case works, especially if
>     zero-copy ever
>      > works for non-TOE.
> 
>     I do not, and I rely on patches I merged upstream to netperf (-a and
>     -A flags)
>     to test it.  I believe there might be some proprietary bits in some
>     FreeBSD
>     downstreams that might make use of this.
> 
>     -- 
>     John Baldwin
> 
> 
> Do you mean these -a and -A flags, or am I looking in the wrong place?
>         -a sizespec
>                Alter the send and receive buffer alignments on the local
>                system.  This defaults to 8 bytes.
> 
>         -A sizespec
>                As -a, but for the remote system.

The aio options are available for the TCP tests but you might have to 
build netperf with a non-default option (that's what I had to do a long 
time back and I haven't checked if it's still needed).

# netperf -t TCP_STREAM -- -h
...
TCP/UDP BSD Sockets Test Options:
     -a                Use aio_write(2)
     -A                Use aio_read(2)
...

# pkg options netperf
netperf - EXAMPLES: off
netperf - EXS: on
netperf - HISTOGRAM: on
netperf - INFO: off
netperf - OMNI: off
netperf - SCTP: on
netperf - SOCKETS: on

Regards,
Navdeep