ix(intel) vs mlxen(mellanox) 10Gb performance

[Rick Macklem]

Daniel Braniss wrote:
> 
> > On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmacklem at uoguelph.ca> wrote:
> > 
> > Daniel Braniss wrote:
> >> 
> >>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron <csforgeron at gmail.com>
> >>> wrote:
> >>> 
> >>> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD
> >>> 10.1. Before 10.1 it was less.
> >>> 
> >> 
> >> this is NOT iperf/3 where i do get close to wire speed,
> >> it’s NFS writes, i.e., almost real work :-)
> >> 
> >>> I used to tweak the card settings, but now it's just stock. You may want
> >>> to
> >>> check your settings, the Mellanox may just have better defaults for your
> >>> switch.
> >>> 
> > Have you tried disabling TSO for the Intel? With TSO enabled, it will be
> > copying
> > every transmitted mbuf chain to a new chain of mbuf clusters via.
> > m_defrag() when
> > TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the
> > 82599 chip
> > these days?)
> > 
> 
> hi Rick
> 
> how can i check the chip?
> 
Haven't a clue. Does "dmesg" tell you? (To be honest, since disabling TSO helped,
I'll bet you don't have a 82598.)

> > This has been fixed in the driver very recently, but those fixes won't be
> > in 10.1.
> > 
> > rick
> > ps: If you could test with 10.2, it would be interesting to see how the ix
> > does with
> >    the current driver fixes in it?
> 
> I new TSO was involved!
> ok, firstly, it’s 10.2 stable.
> with TSO enabled, ix is bad, around 64MGB/s.
> disabling TSO it’s better, around 130
> 
Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c at around
line#2500?
  /* TSO parameters */
2572 	  	         ifp->if_hw_tsomax = 65518;
2573 	  	         ifp->if_hw_tsomaxsegcount = IXGBE_82599_SCATTER;
2574 	  	         ifp->if_hw_tsomaxsegsize = 2048;

They are in stable/10. I didn't look at releng/10.2. (And if they're in a #ifdef
for FreeBSD11, take the #ifdef away.)
If they are there and not ifdef'd, I can't explain why disabling TSO would help.
Once TSO is fixed so that it handles the 64K transmit segments without copying all
the mbufs, I suspect you might get better perf. with it enabled?

Good luck with it, rick

> still, mlxen0 is about 250! with and without TSO
> 
> 
> > 
> >>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw at zxy.spb.ru
> >>> <mailto:slw at zxy.spb.ru>> wrote:
> >>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote:
> >>> 
> >>>> hi,
> >>>>      I have a host (Dell R730) with both cards, connected to an HP8200
> >>>>      switch at 10Gb.
> >>>>      when writing to the same storage (netapp) this is what I get:
> >>>>              ix0:            ~130MGB/s
> >>>>              mlxen0  ~330MGB/s
> >>>>      this is via nfs/tcpv3
> >>>> 
> >>>>      I can get similar (bad) performance with the mellanox if I increase
> >>>>      the file size
> >>>>      to 512MGB.
> >>> 
> >>> Look like mellanox have internal beffer for caching and do ACK
> >>> acclerating.
> >>> 
> >>>>      so at face value, it seems the mlxen does a better use of resources
> >>>>      than the intel.
> >>>>      Any ideas how to improve ix/intel's performance?
> >>> 
> >>> Are you sure about netapp performance?
> >>> _______________________________________________
> >>> freebsd-net at freebsd.org <mailto:freebsd-net at freebsd.org> mailing list
> >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> >>> <https://lists.freebsd.org/mailman/listinfo/freebsd-net>
> >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org
> >>> <mailto:freebsd-net-unsubscribe at freebsd.org>"
> >>> 
> >> 
> >> _______________________________________________
> >> freebsd-stable at freebsd.org mailing list
> >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
> 
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"