ixgbe: Network performance tuning (#TCP connections)

[Alexey Ivanov]

In Linux/Intel I would use following methodology for performance analysis:

Hardware:
* turbostat
  Look for C/P states for cores, frequencies, number of SMIs. [1]
* cpufreq-info
  Look for current driver, frequencies, and governor.
* atop
  Look for interrupt distribution across cores
  Look for context switches, interrupts.
* ethtool
  -S for stats, look for errors, drops, overruns, missed interrupts, etc
  -k for offloads, enable GRO/GSO, rss/rps/rfs/xps[0]
  -g for ring sizes, increase
  -c for interrupt coalescing

Kernel:
* /proc/net/softirq [2] and /proc/interrupts [3]
  Again, distribution, missed, delayed interrupts
  (optional) NUMA-affinity
* perf top
  Look where kernel spends its time.
* iptables
  Look if there are rules (if any) that may affect performance.
* netstat -s, netstat -m
  Look for error counters and buffer counts
* sysctl / grub
  So much to tweak here. Try increasing hashtable sizes, playing with memory buffers and other limits.

BSD has alternatives to most of these, e.g. perf -> hwpmc, ethtool -> ifconfig, softirq -> netisr, menu.lst -> loader.conf


In your case I suppose turning on interrupt coalescing along with enabling hardware offloads may help.


[0] Comparing mutiqueue support Linux vs FreeBSD
	https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/ComparingMutiqueueSupportLinuxvsFreeBSD
[1] You can pin your processor to a specific C-state:
	https://gist.github.com/SaveTheRbtz/f5e8d1ca7b55b6a7897b
[2] You can analyze that data with:
	https://gist.github.com/SaveTheRbtz/172b2e2eb3cbd96b598d
[3] You can set affinity with:
	https://gist.github.com/SaveTheRbtz/8875474

PS. Sorry for so many Linuxisms on freebsd-performance@


> On Feb 5, 2016, at 2:57 PM, Adrian Chadd <adrian.chadd at gmail.com> wrote:
> 
> On 5 February 2016 at 10:15, Meyer, Wolfgang <wolfgang.meyer at hob.de> wrote:
>> 
>> 
>>> -----Original Message-----
>>> From: owner-freebsd-performance at freebsd.org [mailto:owner-freebsd-
>>> performance at freebsd.org] On Behalf Of Allan Jude
>>> Sent: Mittwoch, 3. Februar 2016 22:50
>>> To: freebsd-performance at freebsd.org
>>> Subject: Re: ixgbe: Network performance tuning (#TCP connections)
>>> 
>>> On 2016-02-03 16:34, Adrian Chadd wrote:
>>>> hi,
>>>> 
>>>> can you share your testing program source?
>>>> 
>>>> 
>>>> -a
>>>> 
>>> 
>>> I have a Dual E5-2630 v3 (2x8x 2.40ghz (+HT)) with the Intel X540-AT2 that I
>>> can try to replicate this one to help debug it.
>>> 
>>> --
>>> Allan Jude
>> 
>> I'll try to do some polishing and removal of cruft next week than I hope I will feel comfortable putting it to the public :-)
>> 
>> Not that they are overly sophisticated programs, just some test set-up created in the past and over time one gets used to use it as a sort of private "benchmark".
> 
> Please do - it'd be good to see what you're doing and figure out
> what's causing the poor behaviour.
> 
> Also having more public benchmarks for testing and reproducibility is
> always good. :)
> 
> 
> 
> -adrian
> 
>> 
>> Regards,
>> Wolfgang Meyer
>> 
>> 
>> ________________________________
>> 
>> Follow HOB:
>> 
>> - HOB: http://www.hob.de/redirect/hob.html
>> - Xing: http://www.hob.de/redirect/xing.html
>> - LinkedIn: http://www.hob.de/redirect/linkedin.html
>> - HOBLink Mobile: http://www.hob.de/redirect/hoblinkmobile.html
>> - Facebook: http://www.hob.de/redirect/facebook.html
>> - Twitter: http://www.hob.de/redirect/twitter.html
>> - YouTube: http://www.hob.de/redirect/youtube.html
>> - E-Mail: http://www.hob.de/redirect/mail.html
>> 
>> 
>> HOB GmbH & Co. KG
>> Schwadermuehlstr. 3
>> D-90556 Cadolzburg
>> 
>> Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic
>> 
>> AG Fuerth, HRA 5180
>> Steuer-Nr. 218/163/00107
>> USt-ID-Nr. DE 132747002
>> 
>> Komplementaerin HOB electronic Beteiligungs GmbH
>> AG Fuerth, HRB 3416
>> _______________________________________________
>> freebsd-performance at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-performance
>> To unsubscribe, send any mail to "freebsd-performance-unsubscribe at freebsd.org"
> _______________________________________________
> freebsd-performance at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "freebsd-performance-unsubscribe at freebsd.org"

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20160205/11dbd030/attachment.sig>