Unstable local network throughput

[Ben RUBSON]

> On 11 Aug 2016, at 00:11, Adrian Chadd <adrian.chadd at gmail.com> wrote:
> 
> hi,
> 
> ok, lets start by getting the NUMA bits into the kernel so you can
> mess with things.
> 
> add this to the kernel
> 
> options MAXMEMDOM=8
> (which hopefully is enough)
> options VM_NUMA_ALLOC
> options DEVICE_NUMA
> 
> Then reboot and post your 'dmesg' output to the list. This should show
> exactly which domain devices are in.

http://pastebin.com/raw/yaYEytME

> Install the 'intel-pcm' package. There's a 'pcm-numa.x' command - do
> kldload cpuctl, then run pcm-numa.x and see if it works. It should
> give us some useful information about NUMA.
> (Same as pcm-memory.x, pcm-pcie.x, etc.)

Yes these tools work :

# pcm-numa.x

 Intel(r) Performance Counter Monitor: NUMA monitoring utility 
 Copyright (c) 2009-2016 Intel Corporation

Number of physical cores: 12
Number of logical cores: 24
Number of online logical cores: 24
Threads (logical cores) per physical core: 2
Num sockets: 2
Physical cores per socket: 6
Core PMU (perfmon) version: 3
Number of core PMU generic (programmable) counters: 4
Width of generic (programmable) counters: 48 bits
Number of core PMU fixed counters: 3
Width of fixed counters: 48 bits
Nominal core frequency: 2400000000 Hz
Package thermal spec power: 85 Watt; Package minimum power: 31 Watt; Package maximum power: 170 Watt; 
ERROR: QPI LL monitoring device (0:127:9:2) is missing. The QPI statistics will be incomplete or missing.
Socket 0: 2 memory controllers detected with total number of 5 channels. 1 QPI ports detected.
ERROR: QPI LL monitoring device (0:255:9:2) is missing. The QPI statistics will be incomplete or missing.
Socket 1: 2 memory controllers detected with total number of 5 channels. 1 QPI ports detected.
Socket 0
Max QPI link 0 speed: 16.0 GBytes/second (8.0 GT/second)
Socket 1
Max QPI link 0 speed: 16.0 GBytes/second (8.0 GT/second)

Detected Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz "Intel(r) microarchitecture codename Haswell-EP/EN/EX"
Update every 1.0 seconds
Time elapsed: 1010 ms
Core | IPC  | Instructions | Cycles  |  Local DRAM accesses | Remote DRAM Accesses 
   0   0.70       1158 K     1655 K       577                 245                
   1   0.33        186 K      557 K       160                  15                
   2   0.43        317 K      745 K       385                  31                
   3   0.36        260 K      718 K       232                  33                
   4   0.31        186 K      602 K       188                  11                
   5   0.39        314 K      806 K       371                  43                
   6   0.36        235 K      659 K       257                  46                
   7   0.35        200 K      576 K       133                  44                
   8   0.42        423 K     1011 K       226                  20                
   9   0.60       1309 K     2199 K       379                 104                
  10   0.34        192 K      562 K       161                  26                
  11   0.38        257 K      684 K       158                  44                
  12   0.35        185 K      528 K        39                 121                
  13   0.32        199 K      616 K        51                 171                
  14   0.31        184 K      594 K        34                 130                
  15   0.35        272 K      783 K        47                 256                
  16   0.31        178 K      579 K        26                 127                
  17   0.37        272 K      729 K        87                 204                
  18   0.52        485 K      942 K        35                 204                
  19   0.40        285 K      723 K        16                 147                
  20   0.31        195 K      620 K        10                 134                
  21   0.33        201 K      615 K        30                 114                
  22   0.29        176 K      612 K        24                 110                
  23   0.52        896 K     1716 K        86                 895                
-------------------------------------------------------------------------------------------------------------------
   *   0.43       8575 K       19 M      3712                3275                

> Then next is playing around with interrupt thread / userland cpuset
> and memory affinity. We can look at that next.

Waiting for your instructions !

Ben