High CPU interrupt load on intel I350T4 with igb on 8.3

[John Baldwin]

On Friday, April 26, 2013 7:31:07 am Clément Hermann (nodens) wrote:
> Hi list,
> 
> We use pf+ALTQ for trafic shaping on some routers.
> 
> We are switching to new servers : Dell PowerEdge R620 with 2 8-cores 
> Intel Processor (E5-2650L), 8GB RAM and Intel I350T4 (quad port) using 
> igb driver. The old hardware is using em driver, the CPU load is high 
> but mostly due to kernel and a large pf ruleset.
> 
> On the new hardware, we see high CPU Interrupt load (up to 95%), even 
> though there is not much trafic currently (peaks about 150Mbps and 
> 40Kpps). All queues are used and binded to a cpu according to top, but a 
> lot of CPU time is spent on igb queues (interrupt or wait). The load is 
> fine when we stay below 20Kpps.
> 
> We see no mbuf shortage, no dropped packet, but there is little margin 
> left on CPU time (about 25% idle at best, most of CPU time is spent on 
> interrupts), which is disturbing.
> 
> We have done some tuning, but to no avail :

If you have the processing_limit set to -1, you should never see CPU time 
spent in the igb task threads (any such time means there is a bug).  One such 
bug was fixed in 8.x here (that is after 8.3):

http://svnweb.freebsd.org/base?view=revision&revision=235553

This may not help with any issues in pf(4), but we had workloads at work (not 
involving pf) where this bug could cause boxes to spend 100% CPU in igb 
threads.

-- 
John Baldwin