Sporadic TCP/RST sent to client

[Youssef GHORBAL]

Imagine this set up :

freebsd host port0 <-> switch 1 <-> linux host port0
freebsd host port1 <-> switch 2 <-> linux host port1

On the linux box, port 0&1 are enslaved in a bond with a RR algorithm (Round Robin)
On the freebsd box, port 0&1 are enslaved in a lagg.

switchs 1&2 are configured for doing MLAG.

The Linux box disapatchs packets on both NICs (since the RR algo dictates that) packets are dispatched in order.
Packets outgoing on port0 gets handled by switch1 and hits the freebsd box on port 0
Packets outgoing on port1 gets handled by switch2 and hits the freebsd box on port 1

As I stated earlier, from the tcpdump traces I've done on the freebsd box (both on the lagg interface and the actual ports) packets do arrive ordered but on different NICs and it works great until the elapes times start to be around microsecond.

I don't really have control over the Linux box to make them use other hash algo (but I'm stil trying)

Youssef
------------------------
> On 27 Jun 2017, at 00:13, Matt Joras <matt.joras at gmail.com> wrote:
> 
> Out of curiosity, what sort of lagg setup are you using that's causing
> the TCP packets to be split across the two lagg interfaces?
> 
> Matt
> 
> On Mon, Jun 26, 2017 at 1:35 PM, Navdeep Parhar <nparhar at gmail.com> wrote:
>> On Thu, Jun 22, 2017 at 3:57 PM, Youssef  GHORBAL
>> <youssef.ghorbal at pasteur.fr> wrote:
>>> Hello,
>>> 
>>>        I'm having an issue with a FreeBSD 11 based system, sending sporadically TCP/RST to clients after initial TCP session correctly initiated.
>>>        The sequence goes this way :
>>> 
>>>        1 Client -> Server : SYN
>>>        2 Server -> Client : SYN/ACK
>>>        3 Client -> Server : ACK
>>>        4 Client -> Server : PSH/ACK (upper protocol data sending starts here)
>>>        5 Server -> Client : RST
>>> 
>>>        - The problem happens sporadically, same client and same server can communicate smoothely on the same service port. But from time to time (hours, sometime days) the previous sequence happens.
>>>        - The service running on server is not responsible for the RST sent. The service was deeply profiled and nothing happens to justify the RST.
>>>        - tcpdump on the server side assures that packet arrives timely ordered.
>>>        - the traffic is very light. Some TCP sessions per day.
>>>        - the server is connected using a lagg enslaving two cxgb interfaces.
>>> 
>>>        In my effort to diagnose the problem (try to have a reproductible test case) I noticed that the issue is triggered most likely when those two conditions are met :
>>>        - the ACK (in step 3) and the PSH/ACK (in step 4) arrive on different lagg NICs.
>>>        - the timing between those two packets is sub 10 microseconds.
>>> 
>>>        When searching the interwebs I came across a strangely similar issue reported here 7 years ago :
>>>        https://lists.freebsd.org/pipermail/freebsd-net/2010-August/026029.html
>>> 
>>>        (The OP seemed to have resolved his issue changing the netisr policy from direct to hybrid. but no reference of laggs being used)
>>> 
>>>        I'm pretty sure that I'm hitting some race condition, a scenario where due to multithreading the PSH/ACK is somehow handled before the ACK making the kernel rising TCP/RST since the initial TCP handshake did'nt finish yet.
>>> 
>>>        I've read about netisr work and I was under the impression that even if it's SMP enabled it was made to keep prorocol ordering.
>>> 
>>>        What's the expected behaviour in this scenario on the netisr side ?
>>>        How can I push the investigation further ?
>> 
>> I think you've already figured out the situation here -- the PSH/ACK is likely
>> being handled before the ACK for the SYN because they arrived on different
>> interfaces.  There is nothing in netisr dispatch that will maintain protocol
>> ordering in this case.
>> 
>> Regards,
>> Navdeep
>> _______________________________________________
>> freebsd-net at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"