Re: Increasing TCP TSO size support

What is the link speed that you're working with?

A long time ago, when I worked for a now-defunct 10GbE NIC vendor, I experimented with the benefits of  TSO as we varied the max TSO size.  I cannot recall the platform (it could have been OSX, Solaris, FreeBSD or Linux).  At the time (~2006?) the CPU saving benefits of increasing the max TSO size from 8k to 64k was fairly minimal.    In fact, I seem to recall that there was almost no benefit to TSO sizes larger than 16K.

I was wondering if you see any difference in your benchmark if you cap max TSO size to 8k,  16k,32k, and the default of 64k.  Any change in CPU use, or in your benchmark's performance would be interesting to hear about.

Naively, I'd expect the benchmark performance to remain unchanged until you'd reduced the TSO size so much as to make the host slower than the wire, thereby inserting gaps between TSOs.  That would be reflected in the CPU use as well..

Drew

On Fri, Feb 2, 2024, at 4:21 AM, Scheffenegger, Richard wrote:
> 
> 
> Hi,
> 
> We have run a test for a RPC workload with 1MB IO sizes, and collected the tcp_default_output() len(gth) during the first pass in the output loop.
> 
> In such a scenario, where the application frequently introduces small pauses (since the next large IO is only sent after the corresponding request from the client has been received and processed) between sending additional data, the current TSO limit of 64kB TSO maximum (45*1448 in effect) requires multiple passes in the output routine to send all the allowable (cwnd limited) data.
> 
> I'll try to get a data collection with better granulariy above 90 000 bytes - but even here the average strongly indicates that a majority of transmission opportunities are in the 512 kB area - probably also having to do with LRO and ACK thinning effects by the client.
> 
> With other words, the tcp output has to run about 9 times with TSO, to transmit all elegible data - increasing the FreeBSD supported maximum TSO size to what current hardware could handle (256kB..1MB) would reduce the CPU burden here.
> 
> 
> 
> Is increasing the sofware supported TSO size to allow for what the NICs could nowadays do something anyone apart from us would be interested in (in particular, those who work with the drivers)?
> 
> 
> 
> Best regards,
> 
>   Richard
> 
> 
> 
> 
> 
> 
> 
> tso size (transmissions < 1448 would not be accounted here at all)
> 
>                     # count
> 
> 
> 
> <1000
> 0
> <2000
> 23
> <3000
> 111
> <4000
> 40
> <5000
> 30
> <7000
> 14
> <8000
> 134
> <9000
> 442
> <10000
> 9396
> <20000
> 46227
> <30000
> 25646
> <40000
> 33060
> <60000
> 23162
> <70000
> 24368
> <80000
> 19772
> <90000
> 40101
> >=90000
> 75384169
> Average:
> 578844.44
> 
> *Attachments:*
>  • OpenPGP_0x17BE5899E0B1439B.asc
>  • OpenPGP_signature.asc