ix(intel) vs mlxen(mellanox) 10Gb performance

[Hans Petter Selasky]

On 08/19/15 09:42, Yonghyeon PYUN wrote:
> On Wed, Aug 19, 2015 at 09:00:52AM +0200, Hans Petter Selasky wrote:
>> On 08/18/15 23:54, Rick Macklem wrote:
>>> Ouch! Yes, I now see that the code that counts the # of mbufs is before the
>>> code that adds the tcp/ip header mbuf.
>>>
>>> In my opinion, this should be fixed by setting if_hw_tsomaxsegcount to
>>> whatever
>>> the driver provides - 1. It is not the driver's responsibility to know if
>>> a tcp/ip
>>> header mbuf will be added and is a lot less confusing that expecting the
>>> driver
>>> author to know to subtract one. (I had mistakenly thought that
>>> tcp_output() had
>>> added the tc/ip header mbuf before the loop that counts mbufs in the list.
>>> Btw,
>>> this tcp/ip header mbuf also has leading space for the MAC layer header.)
>>>
>>
>> Hi Rick,
>>
>> Your question is good. With the Mellanox hardware we have separate
>> so-called inline data space for the TCP/IP headers, so if the TCP stack
>> subtracts something, then we would need to add something to the limit,
>> because then the scatter gather list is only used for the data part.
>>
>
> I think all drivers in tree don't subtract 1 for
> if_hw_tsomaxsegcount.  Probably touching Mellanox driver would be
> simpler than fixing all other drivers in tree.
>
>> Maybe it can be controlled by some kind of flag, if all the three TSO
>> limits should include the TCP/IP/ethernet headers too. I'm pretty sure
>> we want both versions.
>>
>
> Hmm, I'm afraid it's already complex.  Drivers have to tell almost
> the same information to both bus_dma(9) and network stack.

Don't forget that not all drivers in the tree set the TSO limits before 
if_attach(), so possibly the subtraction of one TSO fragment needs to go 
into ip_output() ....

--HPS