Unstable local network throughput

[Ben RUBSON]

> On 12 Aug 2016, at 00:52, Adrian Chadd <adrian.chadd at gmail.com> wrote:
> 
> Which ones of these hit the line rate comfortably?

So Adrian, I ran tests again using FreeBSD 11-RC1.
I put iperf throughput in result files (so that we can classify them), as well as top -P ALL and pcm-memory.x.
iperf results : columns 3&4 are for srv1->srv2, columns 5&6 are for srv2->srv1 (both flows running at the same time).



Results, expected throughput (best first) :
11, 01, 05, 07, 06

Results, bad (best first) :
04, 02, 09, 03

Results, worst (best first) :
10, 08



00) Idle system
http://pastebin.com/raw/K1iMVHVF



01) No pinning
http://pastebin.com/raw/7J3HibX0

02) numactl -l fixed-domain-rr -m 0 -c 0
http://pastebin.com/raw/Yt7yYr0K

03) numactl -l fixed-domain-rr -m 0 -c 0
+ cpuset -l <0-11> -x <IRQ>
http://pastebin.com/raw/1FAgDUSU

04) numactl -l fixed-domain-rr -m 0 -c 0
+ cpuset -l <12-23> -x <IRQ>
http://pastebin.com/raw/fTAxrzBb

05) numactl -l fixed-domain-rr -m 1 -c 1
http://pastebin.com/raw/kuAHzKu2

06) numactl -l fixed-domain-rr -m 1 -c 1
+ cpuset -l <0-11> -x <IRQ>
http://pastebin.com/raw/tgtaZgwb

07) numactl -l fixed-domain-rr -m 1 -c 1
+ cpuset -l <12-23> -x <IRQ>
http://pastebin.com/raw/16ReuGFF



08) No pinning, default kernel (no NUMA option)
http://pastebin.com/raw/Ah74fKRx

09) default kernel (no NUMA option)
+ cpuset -l <0-11>
+ cpuset -l <0-11> -x <IRQ>
http://pastebin.com/raw/YE0PxEu8

10) default kernel (no NUMA option)
+ cpuset -l <12-23>
+ cpuset -l <12-23> -x <IRQ>
http://pastebin.com/raw/RPh8aM49



11) No pinning, default kernel (no NUMA option), NUMA BIOS disabled
http://pastebin.com/raw/LyGcLKDd



Ben