ix(intel) vs mlxen(mellanox) 10Gb performance

[Rick Macklem]

Daniel Braniss wrote:
> 
> > On Aug 17, 2015, at 1:41 PM, Christopher Forgeron <csforgeron at gmail.com>
> > wrote:
> > 
> > FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD
> > 10.1. Before 10.1 it was less.
> > 
> 
> this is NOT iperf/3 where i do get close to wire speed,
> it’s NFS writes, i.e., almost real work :-)
> 
> > I used to tweak the card settings, but now it's just stock. You may want to
> > check your settings, the Mellanox may just have better defaults for your
> > switch.
> > 
Have you tried disabling TSO for the Intel? With TSO enabled, it will be copying
every transmitted mbuf chain to a new chain of mbuf clusters via. m_defrag() when
TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the 82599 chip
these days?)

This has been fixed in the driver very recently, but those fixes won't be in 10.1.

rick
ps: If you could test with 10.2, it would be interesting to see how the ix does with
    the current driver fixes in it?

> > On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw at zxy.spb.ru
> > <mailto:slw at zxy.spb.ru>> wrote:
> > On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote:
> > 
> > > hi,
> > >       I have a host (Dell R730) with both cards, connected to an HP8200
> > >       switch at 10Gb.
> > >       when writing to the same storage (netapp) this is what I get:
> > >               ix0:            ~130MGB/s
> > >               mlxen0  ~330MGB/s
> > >       this is via nfs/tcpv3
> > >
> > >       I can get similar (bad) performance with the mellanox if I increase
> > >       the file size
> > >       to 512MGB.
> > 
> > Look like mellanox have internal beffer for caching and do ACK acclerating.
> > 
> > >       so at face value, it seems the mlxen does a better use of resources
> > >       than the intel.
> > >       Any ideas how to improve ix/intel's performance?
> > 
> > Are you sure about netapp performance?
> > _______________________________________________
> > freebsd-net at freebsd.org <mailto:freebsd-net at freebsd.org> mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-net
> > <https://lists.freebsd.org/mailman/listinfo/freebsd-net>
> > To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org
> > <mailto:freebsd-net-unsubscribe at freebsd.org>"
> > 
> 
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"