Re: RFC: NFS trunking (multiple TCP connections for a mount

On Mon, Jun 28, 2021 at 6:24 PM Rick Macklem <rmacklem@uoguelph.ca> wrote:

> The Linux NFS client now has a mount option "nconnect",
> which specifies that multiple TCP connections be created
> for an NFS mount, where RPCs are done on the connections,
> in a round robin fashion. (Alternating between the two TCP
> connections for the case of nconnect=2.)
>
> The Linux man page says:
> nconnect=n
>               When using a connection oriented protocol such as TCP, it
>               may sometimes be advantageous to set up multiple
>               connections between the client and server. For instance,
>               if your clients and/or servers are equipped with multiple
>               network interface cards (NICs), using multiple connections
>               to spread the load may improve overall performance.  In
>               such cases, the nconnect option allows the user to specify
>               the number of connections that should be established
>               between the client and server up to a limit of 16.
>
> I don't understand how multiple TCP connections to the same
> server IP address will distribute the load across multiple network
> interfaces?
> I thought that lagg would have handled this?
>

Splitting a single TCP connection across multiple physical interfaces would
cause tons of out-of-order packets to be received.  The overhead of TCP
reassembly overhead then outweighs the benefit of lagg, especially if there
are multiple connections anyway.  So that's why, unless your physical
interfaces are very slow, it's not worth splitting one TCP connection.
LACP doesn't even allow it.

Something like nconnect might be useful.  But only if there are few NFS
clients anyway.  In my environment there are usually many, so I wouldn't
use nconnect.


>
> I could easily implement this, but I only have low end hardware
> to test on, so I doubt that I will see any performance improvement.
>
> However, I do think that having two TCP connections, where the
> RPCs involving large RPC messages (Read/Readdir/Write) are sent
> on one TCP connection and the RPCs that use small RPC messages
> (Lookup/Access/Getattr,...) are sent on the other one.
> --> This would avoid the frequent small RPCs from getting "logjamed"
>        behind a bunch of large 1Mbyte Read replies, for example.
>
> So, what do you think?
> - Implement "nconnect" with round robin RPC assignment.
> or
> - Implement two TCP connections where large RPCs are done
>   on one and small RPCs on the other.
> or
> ???
>
> I will note I see downsides to doing multiple TCP connections/mount.
> 1 - Uses up more IP port#s.
> 2 - When an NFS server gets overloaded, it will stop receiving RPC
> requests.
>      This will eventually apply backpressure through TCP to the client to
> slow
>      down RPC requests. Having multiple TCP connections would reduce this
>      backpressure effect.
>      --> To be honest, I suspect the slowdown in RPC replies caused by an
>            overloaded server, is more effective feedback to the NFS client
>            than TCP backpressure, but I am not sure.
>
> Comments? rick
>
>
>
>