ix(intel) vs mlxen(mellanox) 10Gb performance

[Rick Macklem]

Daniel Braniss wrote:
> 
> > On Aug 18, 2015, at 12:49 AM, Rick Macklem <rmacklem at uoguelph.ca> wrote:
> > 
> > Daniel Braniss wrote:
> >> 
> >>> On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmacklem at uoguelph.ca> wrote:
> >>> 
> >>> Daniel Braniss wrote:
> >>>> 
> >>>>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron
> >>>>> <csforgeron at gmail.com>
> >>>>> wrote:
> >>>>> 
> >>>>> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD
> >>>>> 10.1. Before 10.1 it was less.
> >>>>> 
> >>>> 
> >>>> this is NOT iperf/3 where i do get close to wire speed,
> >>>> it’s NFS writes, i.e., almost real work :-)
> >>>> 
> >>>>> I used to tweak the card settings, but now it's just stock. You may
> >>>>> want
> >>>>> to
> >>>>> check your settings, the Mellanox may just have better defaults for
> >>>>> your
> >>>>> switch.
> >>>>> 
> >>> Have you tried disabling TSO for the Intel? With TSO enabled, it will be
> >>> copying
> >>> every transmitted mbuf chain to a new chain of mbuf clusters via.
> >>> m_defrag() when
> >>> TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the
> >>> 82599 chip
> >>> these days?)
> >>> 
> >> 
> >> hi Rick
> >> 
> >> how can i check the chip?
> >> 
> > Haven't a clue. Does "dmesg" tell you? (To be honest, since disabling TSO
> > helped,
> > I'll bet you don't have a 82598.)
> > 
> >>> This has been fixed in the driver very recently, but those fixes won't be
> >>> in 10.1.
> >>> 
> >>> rick
> >>> ps: If you could test with 10.2, it would be interesting to see how the
> >>> ix
> >>> does with
> >>>   the current driver fixes in it?
> >> 
> >> I new TSO was involved!
> >> ok, firstly, it’s 10.2 stable.
> >> with TSO enabled, ix is bad, around 64MGB/s.
> >> disabling TSO it’s better, around 130
> >> 
> > Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c at
> > around
> > line#2500?
> >  /* TSO parameters */
> > 2572 	  	         ifp->if_hw_tsomax = 65518;
> > 2573 	  	         ifp->if_hw_tsomaxsegcount = IXGBE_82599_SCATTER;
> > 2574 	  	         ifp->if_hw_tsomaxsegsize = 2048;
> > 
> > They are in stable/10. I didn't look at releng/10.2. (And if they're in a
> > #ifdef
> > for FreeBSD11, take the #ifdef away.)
> > If they are there and not ifdef'd, I can't explain why disabling TSO would
> > help.
> > Once TSO is fixed so that it handles the 64K transmit segments without
> > copying all
> > the mbufs, I suspect you might get better perf. with it enabled?
> > 
> 
> this is 10.2 :
> they are on lines  2509-2511 and I don’t see any #ifdefs around it.
> 
> the plot thickens :-)
> 
If this is just a test machine, maybe you could test with these lines (at about #880)
in sys/netinet/tcp_output.c commented out? (It looks to me like this will disable TSO
for almost all the NFS writes.)
- around line #880 in sys/netinet/tcp_output.c:
			/*
			 * In case there are too many small fragments
			 * don't use TSO:
			 */
			if (len <= max_len) {
				len = max_len;
				sendalot = 1;
				tso = 0;
			}

This was added along with the other stuff that did the if_hw_tsomaxsegcount, etc and I
never noticed it until now (not my patch).

rick

> danny
> 
> > Good luck with it, rick
> > 
> >> still, mlxen0 is about 250! with and without TSO
> >> 
> >> 
> >>> 
> >>>>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <slw at zxy.spb.ru
> >>>>> <mailto:slw at zxy.spb.ru>> wrote:
> >>>>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote:
> >>>>> 
> >>>>>> hi,
> >>>>>>     I have a host (Dell R730) with both cards, connected to an HP8200
> >>>>>>     switch at 10Gb.
> >>>>>>     when writing to the same storage (netapp) this is what I get:
> >>>>>>             ix0:            ~130MGB/s
> >>>>>>             mlxen0  ~330MGB/s
> >>>>>>     this is via nfs/tcpv3
> >>>>>> 
> >>>>>>     I can get similar (bad) performance with the mellanox if I
> >>>>>>     increase
> >>>>>>     the file size
> >>>>>>     to 512MGB.
> >>>>> 
> >>>>> Look like mellanox have internal beffer for caching and do ACK
> >>>>> acclerating.
> >>>>> 
> >>>>>>     so at face value, it seems the mlxen does a better use of
> >>>>>>     resources
> >>>>>>     than the intel.
> >>>>>>     Any ideas how to improve ix/intel's performance?
> >>>>> 
> >>>>> Are you sure about netapp performance?
> >>>>> _______________________________________________
> >>>>> freebsd-net at freebsd.org <mailto:freebsd-net at freebsd.org> mailing list
> >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> >>>>> <https://lists.freebsd.org/mailman/listinfo/freebsd-net>
> >>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org
> >>>>> <mailto:freebsd-net-unsubscribe at freebsd.org>"
> >>>>> 
> >>>> 
> >>>> _______________________________________________
> >>>> freebsd-stable at freebsd.org mailing list
> >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >>>> To unsubscribe, send any mail to
> >>>> "freebsd-stable-unsubscribe at freebsd.org"
> >> 
> >> _______________________________________________
> >> freebsd-stable at freebsd.org mailing list
> >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
> 
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"