Re: Too aggressive TCP ACKs

From: Zhenlei Huang <zlei.huang_at_gmail.com>
Date: Sat, 22 Oct 2022 02:51:32 UTC
> On Oct 22, 2022, at 2:16 AM, Michael Tuexen <michael.tuexen@lurchi.franken.de <mailto:michael.tuexen@lurchi.franken.de>> wrote:
> 
>> On 21. Oct 2022, at 17:00, Zhenlei Huang <zlei.huang@gmail.com <mailto:zlei.huang@gmail.com>> wrote:
>> 
>> 
>>> On Oct 21, 2022, at 10:34 PM, Michael Tuexen <michael.tuexen@lurchi.franken.de <mailto:michael.tuexen@lurchi.franken.de>> wrote:
>>> 
>>>> On 21. Oct 2022, at 16:19, Zhenlei Huang <zlei.huang@gmail.com <mailto:zlei.huang@gmail.com>> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> While I was repeating https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258755 <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258755>, I observed a
>>>> strange behavior. The TCP ACKs from FreeBSD host are too aggressive.
>>>> 
>>>> My setup is simple:
>>>>        A                                 B
>>>>  [ MacOS ]  <====> [ FreeBSD VM ]
>>>> 192.168.120.1            192.168.12.134 (disable tso and lro)
>>>> While A <--- B, i.e. A as server and B as client, the packets rate looks good.
>>>> 
>>>> One session on B:
>>>> 
>>>> root@:~ # iperf3 -c 192.168.120.1 -b 10m
>>>> Connecting to host 192.168.120.1, port 5201
>>>> [  5] local 192.168.120.134 port 54459 connected to 192.168.120.1 port 5201
>>>> [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
>>>> [  5]   0.00-1.00   sec  1.25 MBytes  10.5 Mbits/sec    0    257 KBytes       
>>>> [  5]   1.00-2.00   sec  1.25 MBytes  10.5 Mbits/sec    0    257 KBytes       
>>>> [  5]   2.00-3.00   sec  1.12 MBytes  9.44 Mbits/sec    0    257 KBytes       
>>>> [  5]   3.00-4.00   sec  1.25 MBytes  10.5 Mbits/sec    0    257 KBytes       
>>>> [  5]   4.00-5.00   sec  1.12 MBytes  9.44 Mbits/sec    0    257 KBytes       
>>>> [  5]   5.00-6.00   sec  1.25 MBytes  10.5 Mbits/sec    0    257 KBytes       
>>>> [  5]   6.00-7.00   sec  1.12 MBytes  9.44 Mbits/sec    0    257 KBytes       
>>>> [  5]   7.00-8.00   sec  1.25 MBytes  10.5 Mbits/sec    0    257 KBytes       
>>>> [  5]   8.00-9.00   sec  1.12 MBytes  9.44 Mbits/sec    0    257 KBytes       
>>>> [  5]   9.00-10.00  sec  1.25 MBytes  10.5 Mbits/sec    0    257 KBytes       
>>>> - - - - - - - - - - - - - - - - - - - - - - - - -
>>>> [ ID] Interval           Transfer     Bitrate         Retr
>>>> [  5]   0.00-10.00  sec  12.0 MBytes  10.1 Mbits/sec    0             sender
>>>> [  5]   0.00-10.00  sec  12.0 MBytes  10.1 Mbits/sec                  receiver
>>>> 
>>>> iperf Done.
>>>> 
>>>> Another session on B:
>>>> 
>>>> root@:~ # netstat -w 1 -I vmx0
>>>>           input           vmx0           output
>>>>  packets  errs idrops      bytes    packets  errs      bytes colls
>>>>        0     0     0          0          0     0          0     0
>>>>        0     0     0          0          0     0          0     0
>>>>      342     0     0      22600        526     0     775724     0
>>>>      150     0     0       9900        851     0    1281454     0
>>>>      109     0     0       7194        901     0    1357850     0
>>>>      126     0     0       8316        828     0    1246632     0
>>>>      122     0     0       8052        910     0    1370780     0
>>>>      109     0     0       7194        819     0    1233702     0
>>>>      120     0     0       7920        910     0    1370780     0
>>>>      110     0     0       7260        819     0    1233702     0
>>>>      123     0     0       8118        910     0    1370780     0
>>>>      109     0     0       7194        819     0    1233702     0
>>>>       73     0     0       5088        465     0     686342     0
>>>>        0     0     0          0          0     0          0     0
>>>>        0     0     0          0          0     0          0     0
>>>> 
>>>> 
>>>> 
>>>> ================================================================
>>>> 
>>>> 
>>>> While A ---> B, i.e. A as client and B as server, the ACKs sent from B looks strange.
>>>> 
>>>> Session on A:
>>>> 
>>>> % iperf3 -c 192.168.120.134 -b 10m
>>>> Connecting to host 192.168.120.134, port 5201
>>>> [  5] local 192.168.120.1 port 52370 connected to 192.168.120.134 port 5201
>>>> [ ID] Interval           Transfer     Bitrate
>>>> [  5]   0.00-1.00   sec  1.25 MBytes  10.5 Mbits/sec                  
>>>> [  5]   1.00-2.00   sec  1.25 MBytes  10.5 Mbits/sec                  
>>>> [  5]   2.00-3.00   sec  1.12 MBytes  9.44 Mbits/sec                  
>>>> [  5]   3.00-4.00   sec  1.25 MBytes  10.5 Mbits/sec                  
>>>> [  5]   4.00-5.00   sec  1.12 MBytes  9.44 Mbits/sec                  
>>>> [  5]   5.00-6.00   sec  1.25 MBytes  10.5 Mbits/sec                  
>>>> [  5]   6.00-7.00   sec  1.12 MBytes  9.44 Mbits/sec                  
>>>> [  5]   7.00-8.00   sec  1.25 MBytes  10.5 Mbits/sec                  
>>>> [  5]   8.00-9.00   sec  1.12 MBytes  9.44 Mbits/sec                  
>>>> [  5]   9.00-10.00  sec  1.25 MBytes  10.5 Mbits/sec                  
>>>> - - - - - - - - - - - - - - - - - - - - - - - - -
>>>> [ ID] Interval           Transfer     Bitrate
>>>> [  5]   0.00-10.00  sec  12.0 MBytes  10.1 Mbits/sec                  sender
>>>> [  5]   0.00-10.00  sec  12.0 MBytes  10.1 Mbits/sec                  receiver
>>>> 
>>>> iperf Done.
>>>> 
>>>> Session on B:
>>>> 
>>>> root@:~ # netstat -w 1 -I vmx0
>>>>           input           vmx0           output
>>>>  packets  errs idrops      bytes    packets  errs      bytes colls
>>>>        0     0     0          0          0     0          0     0
>>>>        0     0     0          0          0     0          0     0
>>>>      649     0     0     960562        330     0      21800     0
>>>>      819     0     0    1233702        415     0      27390     0
>>>>      910     0     0    1370780        459     0      30294     0
>>>>      819     0     0    1233702        415     0      27390     0
>>>>      910     0     0    1370780        459     0      30294     0
>>>>      910     0     0    1370780        460     0      30360     0
>>>>      819     0     0    1233702        414     0      27324     0
>>>>      910     0     0    1370780        460     0      30360     0
>>>>      819     0     0    1233702        414     0      27324     0
>>>>      910     0     0    1370780        460     0      30360     0
>>>>      285     0     0     412287        147     0       9981     0
>>>>        0     0     0          0          0     0          0     0
>>>>        0     0     0          0          0     0          0     0
>>>>        0     0     0          0          0     0          0     0
>>>> 
>>>> 
>>>> The ACK packets replied from B (the FreeBSD VM) are too aggressive. They are
>>>> about one half of TCP packets received from A.
>>>> 
>>>> I've tested with different bitrates, from 10m to 300m, all behave the same.
>>>> Tested with baremetal FreeBSD 13.1 Box as B (with intel em driver), the 
>>>> bitrates is 1g, also  behaves the same.
>>>> 
>>>> Also tried different FreeBSD versions, 11.4, 12.3, stable/13 and current/14 all 
>>>> behave the same.
>>>> 
>>>> 
>>>> My question is, is that the expected behavior of current default TCP stack?
>>> That is what I would expect. TCP (on FreeBSD) is acking every other packet. This
>>> is also what is specified. MacOS, at least newer versions, send less ACKs.
>> Thanks for fast response!
>> 
>> My have old memories about SACK which helps TCP performance. This behavior
>> seems odd from my mind. But those memories date back to 2008, that is 14 years ago.
> I don't think anything has changed since then from a specification point of view
Hacked some RFCs from 1122, and the transport protocol is stable, and apparently
it should be.
>> 
>> The current implementation of TCP stack in FreeBSD head is too complexed for me.
>> Can you please point me the RFCs specifying this? So I can start over with a quick glue.
> Send an ACK for every other frame if everything is OK, send it immediately if there are some gaps:
> https://datatracker.ietf.org/doc/html/rfc9293#section-3.8.6.3 <https://datatracker.ietf.org/doc/html/rfc9293#section-3.8.6.3>
> This applies also to the case where you use SACK.
I think I confused SACK with delayed ACK.

Thanks!
> 
> Best regards
> Michael
>> 
>> Thanks!
>>> 
>>> Best regards
>>> Michael
>>>> 
>>>> 
>>>> 
>>>> Best regards,
>>>> Zhenlei