svn commit: r327559 - in head: . sys/net

[Adrian Chadd]

does it also happen when you actually enable RSS in the kernel? Since
like I went through a whole lot of pain to assign a flowid at
connection setup time.



-a


On 4 January 2018 at 15:37, Steven Hartland <steven at multiplay.co.uk> wrote:
>
>
> On 04/01/2018 22:42, hiren panchasara wrote:
>
> On 01/04/18 at 09:52P, Steven Hartland wrote:
>
> On 04/01/2018 20:50, Eugene Grosbein wrote:
>
> 05.01.2018 3:05, Steven Hartland wrote:
>
> Author: smh
> Date: Thu Jan  4 20:05:47 2018
> New Revision: 327559
> URL: https://svnweb.freebsd.org/changeset/base/327559
>
> Log:
>    Disabled the use of flowid for lagg by default
>
>    Disabled the use of RSS hash from the network card aka flowid for
>    lagg(4) interfaces by default as it's currently incompatible with
>    the lacp and loadbalance protocols.
>
>    The incompatibility is due to the fact that the flowid isn't know
>    for the first packet of a new outbound stream which can result in
>    the hash calculation method changing and hence a stream being
>    incorrectly split across multiple interfaces during normal
>    operation.
>
>    This can be re-enabled by setting the following in loader.conf:
>    net.link.lagg.default_use_flowid="1"
>
>    Discussed with: kmacy
>    Sponsored by:	Multiplay
>
> RSS by definition has meaning to received stream. What is "outbound" stream
> in this context, why can the hash calculatiom method change and what exactly
> does it mean "a stream being incorrectly split"?
>
> Yes RSS is indeed a received stream but that is used by lagg for lacp
> and loadbalance protocols to decide which port of the lagg to "send" the
> packet out of. As the flowid is not known when a new "output" stream is
> instigated the current code falls back to manual hash calculation to
> determine which port to send the initial packet from. Once a response is
> received a tx then uses the flowid. This change of hash calculation
> method can result in the initial packet being sent from a different port
> than the rest of the stream; this is what I meant by "incorrectly split".
>
> For my understanding, is this just an issue for the first packet when we
> originate the flow? Once we have a response and if flowid is there, we'd
> use it, right? OR am I missing something?
>
> Initially yes, but that can cause a whole cascading set of problems. If the
> source machine sends from two different ports then flow can traverse across
> the network using different paths and hence arrive at the destination on
> different ports too, causing the corresponding  issue on the other side.
>
> And with this change, we'd always go and do manual calculation even when
> we have a valid flowid (i.e. we didn't initiate a connection)?
>
> Correct, but there's potentially no easy way to correctly determine what the
> flowid and hence hash should be in this case, likely impossible if the lagg
> consists of different interface types.
>
> In addition if the hardware hash doesn't match the requested one as per
> laggproto then additional issues could also be triggered.
>
> Our TCP stack seems fragile during setup to out of order packets which this
> multipath behavior causes, we've seen this on our loadbalancers which is
> what triggered the investigation. The concrete result is many aborted TCP
> connections, over 300k ~2% on the machine I'm looking at.
>
> I hope there's some improvements that can be made, for example if we can
> determine the stream was instigated remotely then flowid would always be
> valid hence we can use it assuming it matches the requested spec or if we
> can make it clear to the user that laggproto is not the one they requested,
> I'm open to ideas?
>
>     Regards
>     Steve
>