svn commit: r271946 - in head/sys: dev/oce dev/vmware/vmxnet3 dev/xen/netfront kern net netinet ofed/drivers/net/mlx4 sys

[Adrian Chadd]

On 15 September 2015 at 04:16, Hans Petter Selasky <hps at selasky.org> wrote:
> On 09/14/15 20:35, Adrian Chadd wrote:
>>
>> Hi,
>>
>> So what's the actual behaviour of the new tso logic before and after
>> the above change in tsomax?
>
>
> Hi,
>
> The behaviour is the same, only the limits have changed a bit.
>
>> like, what are the actual packet sizes
>>
>> being sent up to the hardware?
>
>
> It is not about the packet sizes, it is about the number of packets we
> transmit per TSO block.
>
>> Is TSO or the TCP stack so fragile that
>> a slight change in how packets are broken up results in ridiculously
>> less throughput? It's only a few bytes.
>
>
> Network adapters which support TSO typically has a hard limit on the number
> of mbufs it can load. When we exceed this limit, m_defrag() or packet drop
> is next. It is the responsibility of the TCP stack to generated mbuf chains
> which are within the network adapter given limits. Previously only the
> length was accounted for. Now we also account for the number of segments,
> because there are many ways a 64K bytes long mbuf chain can be generated.

I know all of this. What I'm asking is different - what about the
change(s) that broke/fixed the Xen network performance actually caused
the change in behaviour? I know things are sensitive to how the mbufs
are broken up - I'd like to see exactly this.

Whilst futzing around at Netflix on TCP behaviour, I found with
everything lined up correctly I'd get 40gbit TCP over a lot of sockets
on sandy bridge hardware - but only as long as the TCP send buffer
size resulted in nicely 4k segments. Some (larger) send buffer sizes
resulted in the stack settling on non-4k segments for whatever reason,
which drastically changed TSO performance. So, subtle changes had much
bigger effects. I'm asking what specifically is going on here.

+1 on the "going to make things more correct" path, but -2 on the
"subtle changes and big unintended, un-described effects" results :(


-adrian