Sporadic TCP/RST sent to client

[Youssef GHORBAL]

> On 27 Jun 2017, at 12:54, sthaug at nethelp.no wrote:
> 
>> Imagine this set up :
>> 
>> freebsd host port0 <-> switch 1 <-> linux host port0
>> freebsd host port1 <-> switch 2 <-> linux host port1
>> 
>> On the linux box, port 0&1 are enslaved in a bond with a RR algorithm (Round Robin)
>> On the freebsd box, port 0&1 are enslaved in a lagg.
>> 
>> switchs 1&2 are configured for doing MLAG.
>> 
>> The Linux box disapatchs packets on both NICs (since the RR algo dictates that) packets are dispatched in order.
>> Packets outgoing on port0 gets handled by switch1 and hits the freebsd box on port 0
>> Packets outgoing on port1 gets handled by switch2 and hits the freebsd box on port 1
>> 
>> As I stated earlier, from the tcpdump traces I've done on the freebsd box (both on the lagg interface and the actual ports) packets do arrive ordered but on different NICs and it works great until the elapes times start to be around microsecond.
>> 
>> I don't really have control over the Linux box to make them use other hash algo (but I'm stil trying)
> 
> If the Linux box is using round robin you shouldn't expect to be able
> to "fix" the problem at the FreeBSD end.

There is nothing in the 802.3ad that mandates stickiness of flows per NIC, the only thing explicit is that hash algorithm needs to maintain packet order. In this case, strictly speaking, it's : Packets do leave in "order" and do arrive in "order".

> On routers and switches (which is what I normally work with) the hash
> algorithm used for LAG connections ensures that one "flow" always uses
> the same path, thus no reordering. A typical hash algorithm uses a
> 5-tuple with (src ip, src port, dst ip, dst port, protocol) as input.
> 
> So the advice in this case is simple - don't use round robin! Yes, I
> understand you don't control the Linux box.


Sure, I was just wondering if the FreeBSD network stack was built with the fact that each flow needs to arrive on the same NIC and the system was designed with this assumption in mind or not.

I reported it here, thinking that maybe it's a subtle buggy corner case and maybe the community was interesting to know about and maybe fix :

- If the stack is working as expected and was built with the assumption that each incoming flow needs to stick to a NIC during it's lifetime, maybe documentation needs to be more explicit regarding this situation. In that case I'll file documentation enhancement bug report.
- If the stack is misbehaving, maybe help the community identify the root cause and help fixing it

Youssef