High CPU interrupt load on intel I350T4 with igb on 8.3

[Barney Cordoba]

You have to admit there's a problem before you can fix it. If Eugene is 
going to blame to bottleneck and no one is going to tell him he's wrong,
then there is no discussion.

The solution in this case is to use 1 queue, which was my suggestion
many days ago. 

The defaults are broken. The driver should default to 1 queue, and be
tuned to the system environment. With 2 NICs in the box, the defaults
are defective. 

1 queue should always work. Other settings require tuning and an
understanding of how things work. 

I've had to support i350 so I've been playing with the driver a bit. It 
works fine with lots of cores. But you have to have more cores than
queues. 2 cards with 4 queues on a 6 physical core system gets into a
contention problem at certain loads.

I've also removed the cpu bindings, which is about all I'm free to disclose.

The driver needs a tuning doc as much as anything else.

BC




--- On Sat, 5/11/13, Adrian Chadd <adrian at freebsd.org> wrote:

> From: Adrian Chadd <adrian at freebsd.org>
> Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3
> To: "Hooman Fazaeli" <hoomanfazaeli at gmail.com>
> Cc: "Barney Cordoba" <barney_cordoba at yahoo.com>, ""Clément Hermann (nodens)"" <nodens2099 at gmail.com>, "Eugene Grosbein" <egrosbein at rdtc.ru>, freebsd-net at freebsd.org
> Date: Saturday, May 11, 2013, 6:16 PM
> Hi,
> 
> The motivation behind the locking scheme in igb in friends
> is for a
> very specific, userland-traffic-origin workload.
> 
> Sure, it may or may not work well for forwarding/filtering
> workloads.
> 
> If you want to fix it, let's have a discussion about how to
> do it,
> followed by some patches to do so.
> 
> 
> 
> 
> Adrian
> 
> On 11 May 2013 13:12, Hooman Fazaeli <hoomanfazaeli at gmail.com>
> wrote:
> > On 5/11/2013 8:26 PM, Barney Cordoba wrote:
> >> Clearly you don't understand the problem. Your
> logic is that because other drivers are defective also;
> therefore its not a driver problem? The problem is caused by
> a multi-threaded driver that
> >> haphazardly launches tasks and that doesn't manage
> the case that the rest of the system can't handle the load.
> It's no different than a driver that barfs when mbuf
> clusters are exhausted. The answer
> >> isn't to increase memory or mbufs, even though that
> may alleviate the problem. The answer is to fix the driver,
> so that it doesn't crash the system for an event that is
> wholly predictable. igb has
> >> 1) too many locks and 2) exasperates the problem by
> binding to cpus, which causes it to not only have to wait
> for the lock to free, but also for a specific cpu to become
> free. So it chugs along
> >> happily until it encounters a bottleneck, at which
> point it quickly blows up the entire system in a domino
> effect. It needs to manage locks more efficiently, and also
> to detect when the backup is
> >> unmanageable. Ever since FreeBSD 5 the answer has
> been "it's fixed in 7, or its fixed in 9, or it's fixed in
> 10". There will always be bottlenecks, and no driver should
> blow up the system no matter
> >> what intermediate code may present a problem. Its
> the driver's responsibility to behave and to drop packets if
> necessary. BC
> >
> > And how the driver should behave? You suggest dropping
> the packets. Even if we accept
> > that dropping packets is a good strategy in all
> configurations (which I doubt), the driver is
> > definitely not the best place to implement it, since
> that involves duplication of similar
> > code between drivers. Somewhere like the Ethernet layer
> is a much better choice to watch
> > load of packets and drop them to prevent them to eat
> all the cores. Furthermore, ignoring
> > the fact that pf is not optimized for multi-processors
> and blaming drivers for not adjusting
> > themselves with the this pf's fault, is a bit unfair, I
> believe.
> >
> >
> > --
> >
> > Best regards.
> > Hooman Fazaeli
> >
> > _______________________________________________
> > freebsd-net at freebsd.org
> mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
> _______________________________________________
> freebsd-net at freebsd.org
> mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>