High interrupt rate on a PF box + performance

[Jeremy Chadwick]

On Thu, Jan 27, 2011 at 08:39:40PM +0100, Damien Fleuriot wrote:
> 
> 
> On 1/27/11 7:46 PM, Sergey Lobanov wrote:
> > В сообщении от Пятница 28 января 2011 00:55:35 автор Damien Fleuriot написал:
> >> On 1/27/11 6:41 PM, Vogel, Jack wrote:
> >>> Jeremy is right, if you have a problem the first step is to try the
> >>> latest code.
> >>>
> >>> However, when I look at the interrupts below I don't see what the problem
> >>> is? The Broadcom seems to have about the same rate, it just doesn't have
> >>> MSIX (multiple vectors).
> >>>
> >>> Jack
> >>
> >> My main concern is that the CPU %interrupt is quite high, also, we seem
> >> to be experiencing input errors on the interfaces.
> > Would you show igb tuning which is done in loader.conf and output of sysctl 
> > dev.igb.0?
> > Did you rise number of igb descriptors such as:
> > hw.igb.rxd=4096
> > hw.igb.txd=4096 ?
> 
> There is no tuning at all on our part in the loader's conf.
> 
> Find below the sysctls:
> 
> # sysctl -a |grep igb
> dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 1.7.3
> dev.igb.0.%driver: igb
> dev.igb.0.%location: slot=0 function=0
> dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10d6 subvendor=0x8086
> subdevice=0x145a class=0x020000
> dev.igb.0.%parent: pci14
> dev.igb.0.debug: -1
> dev.igb.0.stats: -1
> dev.igb.0.flow_control: 3
> dev.igb.0.enable_aim: 1
> dev.igb.0.low_latency: 128
> dev.igb.0.ave_latency: 450
> dev.igb.0.bulk_latency: 1200
> dev.igb.0.rx_processing_limit: 100
> dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 1.7.3
> dev.igb.1.%driver: igb
> dev.igb.1.%location: slot=0 function=1
> dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10d6 subvendor=0x8086
> subdevice=0x145a class=0x020000
> dev.igb.1.%parent: pci14
> dev.igb.1.debug: -1
> dev.igb.1.stats: -1
> dev.igb.1.flow_control: 3
> dev.igb.1.enable_aim: 1
> dev.igb.1.low_latency: 128
> dev.igb.1.ave_latency: 450
> dev.igb.1.bulk_latency: 1200
> dev.igb.1.rx_processing_limit: 100

I'm not aware of how to tune igb(4), so the advice Sergey gave you may
be applicable.  You'll need to schedule downtime to adjust those
tunables however (since a reboot will be requried).

I also reviewed the munin graphs.  I don't see anything necessarily
wrong.  However, you omitted yearly graphs for the network interfaces.
Why I care about that:

The pf state table (yearly) graph basically correlates with the CPU
usage (yearly) graph, and I expect that the yearly network graphs would
show a similar trend: an increase in your overall traffic over the
course of a year.

What I'm trying to figure out is what you're concerned about.  You are
in fact pushing anywhere between 60-120MBytes/sec across these
interfaces.  Given those numbers, I'm not surprised by the ""high""
interrupt usage.

Graphs of this nature usually indicate that you're hitting a
"bottleneck" (for lack of better word) where you're simply doing "too
much" with a single machine (given its network throughput).  The machine
is spending a tremendous amount of CPU time handling network traffic,
and equally as much with regards to the pf usage.

If you want my opinion based on the information I have so far, it's
this: you need to scale your infrastructure.  You can no longer rely on
a single machine to handle this amount of traffic.

As for the network errors you see -- to get low-level NIC and driver
statistics, you'll need to run "sysctl dev.igb.X.stats=1" then run
"dmesg" and look at the numbers shown (the sysctl command won't output
anything itself).  This may help indicate where the packets are being
lost.  You should also check the interface counters on the switch which
these interfaces are connected to.  I sure hope it's a managed switch
which can give you those statistics.

Hope this helps, or at least acts as food for thought.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |