ix(intel) vs mlxen(mellanox) 10Gb performance

[Daniel Braniss]

> On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmacklem at uoguelph.ca> wrote:
> 
> Daniel Braniss wrote:
>> 
>>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron <csforgeron at gmail.com>
>>> wrote:
>>> 
>>> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD
>>> 10.1. Before 10.1 it was less.
>>> 
>> 
>> this is NOT iperf/3 where i do get close to wire speed,
>> it’s NFS writes, i.e., almost real work :-)
>> 
>>> I used to tweak the card settings, but now it's just stock. You may want to
>>> check your settings, the Mellanox may just have better defaults for your
>>> switch.
>>> 
> Have you tried disabling TSO for the Intel? With TSO enabled, it will be copying
> every transmitted mbuf chain to a new chain of mbuf clusters via. m_defrag() when
> TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the 82599 chip
> these days?)
> 

hi Rick

how can i check the chip?

> This has been fixed in the driver very recently, but those fixes won't be in 10.1.
> 
> rick
> ps: If you could test with 10.2, it would be interesting to see how the ix does with
>    the current driver fixes in it?

I new TSO was involved! 
ok, firstly, it’s 10.2 stable.
with TSO enabled, ix is bad, around 64MGB/s.
disabling TSO it’s better, around 130

still, mlxen0 is about 250! with and without TSO


> 
>>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw at zxy.spb.ru
>>> <mailto:slw at zxy.spb.ru>> wrote:
>>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote:
>>> 
>>>> hi,
>>>>      I have a host (Dell R730) with both cards, connected to an HP8200
>>>>      switch at 10Gb.
>>>>      when writing to the same storage (netapp) this is what I get:
>>>>              ix0:            ~130MGB/s
>>>>              mlxen0  ~330MGB/s
>>>>      this is via nfs/tcpv3
>>>> 
>>>>      I can get similar (bad) performance with the mellanox if I increase
>>>>      the file size
>>>>      to 512MGB.
>>> 
>>> Look like mellanox have internal beffer for caching and do ACK acclerating.
>>> 
>>>>      so at face value, it seems the mlxen does a better use of resources
>>>>      than the intel.
>>>>      Any ideas how to improve ix/intel's performance?
>>> 
>>> Are you sure about netapp performance?
>>> _______________________________________________
>>> freebsd-net at freebsd.org <mailto:freebsd-net at freebsd.org> mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> <https://lists.freebsd.org/mailman/listinfo/freebsd-net>
>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org
>>> <mailto:freebsd-net-unsubscribe at freebsd.org>"
>>> 
>> 
>> _______________________________________________
>> freebsd-stable at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"