Sporadic TCP/RST sent to client

[Matt Joras]

On Tue, Jun 27, 2017 at 5:05 AM, Youssef  GHORBAL
<youssef.ghorbal at pasteur.fr> wrote:
>
> There is nothing in the 802.3ad that mandates stickiness of flows per NIC, the only thing explicit is that hash algorithm needs to maintain packet order. In this case, strictly speaking, it's : Packets do leave in "order" and do arrive in "order".
I think the important point is that the ordering is not guaranteed in
this case, despite whether it's happening or not. As soon as you are
using a round-robin lagg on one end you've pretty much lost all
guarantees of ordering at the remote end. Unless the switch has some
way to know, which as Steinar noted is usually done through a
negotiated or statically-configured hash-based lagg, there's no way
for it to enforce the ordering you're expecting for proper behaviour.
So even if there was some notion of protocol ordering in netisr, the
fact that you're using round-robin on one endpoint opens up the
possibility for this kind of situation anyway.

Further, I would argue that round robin is not a valid 802.3ad/802.1AX
algorithm, per how it defines a frame distributor:

"This standard does not mandate any particular distribution
algorithm(s); however, any distribution algorithm shall ensure that,
when frames are received by a Frame Collector as specified in 5.2.3,
the algorithm shall not cause:
a) Misordering of frames that are part of any given conversation, or
b) Duplication of frames.

The above requirement to maintain frame ordering is met by ensuring
that all frames that compose a given conversation are transmitted on a
single link in the order that they are generated by the MAC Client;
hence, this requirement does not involve the addition (or
modification) of any information to the MAC frame, nor any buffering
or processing on the part of the corresponding Frame Collector in
order to reorder frames."

> Sure, I was just wondering if the FreeBSD network stack was built with the fact that each flow needs to arrive on the same NIC and the system was designed with this assumption in mind or not.
>
> I reported it here, thinking that maybe it's a subtle buggy corner case and maybe the community was interesting to know about and maybe fix :
>
> - If the stack is working as expected and was built with the assumption that each incoming flow needs to stick to a NIC during it's lifetime, maybe documentation needs to be more explicit regarding this situation. In that case I'll file documentation enhancement bug report.
> - If the stack is misbehaving, maybe help the community identify the root cause and help fixing it
>
As far as I can tell, as Navdeep noted, there's no unexpected
behaviour in your case. "Flows" are a concept that the protocols, in
this case TCP, knows about. The devices themselves (Ethernet cards)
usually have mechanics to make packet delivery decisions based on flow
information (e.g. RSS hashing), but as far as I know that is generally
limited within a single port, so it doesn't really help in the general
case of a lagg.

Matt