High CPU interrupt load on intel I350T4 with igb on 8.3

[Barney Cordoba]

--- On Thu, 5/9/13, Eugene Grosbein <egrosbein at rdtc.ru> wrote:

> From: Eugene Grosbein <egrosbein at rdtc.ru>
> Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3
> To: ""Clément Hermann (nodens)"" <nodens2099 at gmail.com>
> Cc: freebsd-net at freebsd.org
> Date: Thursday, May 9, 2013, 10:55 AM
> On 26.04.2013 18:31, "Clément
> Hermann (nodens)" wrote:
> > Hi list,
> > 
> > We use pf+ALTQ for trafic shaping on some routers.
> > 
> > We are switching to new servers : Dell PowerEdge R620
> with 2 8-cores 
> > Intel Processor (E5-2650L), 8GB RAM and Intel I350T4
> (quad port) using 
> > igb driver. The old hardware is using em driver, the
> CPU load is high 
> > but mostly due to kernel and a large pf ruleset.
> > 
> > On the new hardware, we see high CPU Interrupt load (up
> to 95%), even 
> > though there is not much trafic currently (peaks about
> 150Mbps and 
> > 40Kpps). All queues are used and binded to a cpu
> according to top, but a 
> > lot of CPU time is spent on igb queues (interrupt or
> wait). The load is 
> > fine when we stay below 20Kpps.
> > 
> > We see no mbuf shortage, no dropped packet, but there
> is little margin 
> > left on CPU time (about 25% idle at best, most of CPU
> time is spent on 
> > interrupts), which is disturbing.
> 
> It seems you suffer from pf lock contention. You should stop
> using pf
> with multi-core systems with 8.3. Move to ipfw+dummynet or
> ng_car for 8.3
> or move to 10.0-CURRENT having new, rewritten pf that does
> not have this problem.
> 
> Network device driver is not guilty here, that's just pf's
> contention
> running in igb's context.
> 
> Eugene Grosbein

They're both at play. Single threadedness aggravates subsystems that 
have too many lock points.

It can also be "solved" with using 1 queue, because then you don't
have 4 queues going into a single thread.

BC