ix(intel) vs mlxen(mellanox) 10Gb performance

[Daniel Braniss]

> On Aug 18, 2015, at 12:49 AM, Rick Macklem <rmacklem at uoguelph.ca> wrote:
> 
> Daniel Braniss wrote:
>> 
>>> On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmacklem at uoguelph.ca> wrote:
>>> 
>>> Daniel Braniss wrote:
>>>> 
>>>>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron <csforgeron at gmail.com>
>>>>> wrote:
>>>>> 
>>>>> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD
>>>>> 10.1. Before 10.1 it was less.
>>>>> 
>>>> 
>>>> this is NOT iperf/3 where i do get close to wire speed,
>>>> it’s NFS writes, i.e., almost real work :-)
>>>> 
>>>>> I used to tweak the card settings, but now it's just stock. You may want
>>>>> to
>>>>> check your settings, the Mellanox may just have better defaults for your
>>>>> switch.
>>>>> 
>>> Have you tried disabling TSO for the Intel? With TSO enabled, it will be
>>> copying
>>> every transmitted mbuf chain to a new chain of mbuf clusters via.
>>> m_defrag() when
>>> TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the
>>> 82599 chip
>>> these days?)
>>> 
>> 
>> hi Rick
>> 
>> how can i check the chip?
>> 
> Haven't a clue. Does "dmesg" tell you? (To be honest, since disabling TSO helped,
> I'll bet you don't have a 82598.)
> 
>>> This has been fixed in the driver very recently, but those fixes won't be
>>> in 10.1.
>>> 
>>> rick
>>> ps: If you could test with 10.2, it would be interesting to see how the ix
>>> does with
>>>   the current driver fixes in it?
>> 
>> I new TSO was involved!
>> ok, firstly, it’s 10.2 stable.
>> with TSO enabled, ix is bad, around 64MGB/s.
>> disabling TSO it’s better, around 130
>> 
> Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c at around
> line#2500?
>  /* TSO parameters */
> 2572 	  	         ifp->if_hw_tsomax = 65518;
> 2573 	  	         ifp->if_hw_tsomaxsegcount = IXGBE_82599_SCATTER;
> 2574 	  	         ifp->if_hw_tsomaxsegsize = 2048;
> 
> They are in stable/10. I didn't look at releng/10.2. (And if they're in a #ifdef
> for FreeBSD11, take the #ifdef away.)
> If they are there and not ifdef'd, I can't explain why disabling TSO would help.
> Once TSO is fixed so that it handles the 64K transmit segments without copying all
> the mbufs, I suspect you might get better perf. with it enabled?
> 

this is 10.2 :
they are on lines  2509-2511 and I don’t see any #ifdefs around it.

the plot thickens :-)

danny

> Good luck with it, rick
> 
>> still, mlxen0 is about 250! with and without TSO
>> 
>> 
>>> 
>>>>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw at zxy.spb.ru
>>>>> <mailto:slw at zxy.spb.ru>> wrote:
>>>>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote:
>>>>> 
>>>>>> hi,
>>>>>>     I have a host (Dell R730) with both cards, connected to an HP8200
>>>>>>     switch at 10Gb.
>>>>>>     when writing to the same storage (netapp) this is what I get:
>>>>>>             ix0:            ~130MGB/s
>>>>>>             mlxen0  ~330MGB/s
>>>>>>     this is via nfs/tcpv3
>>>>>> 
>>>>>>     I can get similar (bad) performance with the mellanox if I increase
>>>>>>     the file size
>>>>>>     to 512MGB.
>>>>> 
>>>>> Look like mellanox have internal beffer for caching and do ACK
>>>>> acclerating.
>>>>> 
>>>>>>     so at face value, it seems the mlxen does a better use of resources
>>>>>>     than the intel.
>>>>>>     Any ideas how to improve ix/intel's performance?
>>>>> 
>>>>> Are you sure about netapp performance?
>>>>> _______________________________________________
>>>>> freebsd-net at freebsd.org <mailto:freebsd-net at freebsd.org> mailing list
>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>> <https://lists.freebsd.org/mailman/listinfo/freebsd-net>
>>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org
>>>>> <mailto:freebsd-net-unsubscribe at freebsd.org>"
>>>>> 
>>>> 
>>>> _______________________________________________
>>>> freebsd-stable at freebsd.org mailing list
>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
>>>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
>> 
>> _______________________________________________
>> freebsd-stable at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"