igb interrupt moderation

[Barney Cordoba]

--- On Sun, 1/3/10, Michael Tüxen <Michael.Tuexen at lurchi.franken.de> wrote:

> From: Michael Tüxen <Michael.Tuexen at lurchi.franken.de>
> Subject: Re: igb interrupt moderation
> To: "Barney Cordoba" <barney_cordoba at yahoo.com>
> Cc: "Mike Tancsa" <mike at sentex.net>, freebsd-net at freebsd.org
> Date: Sunday, January 3, 2010, 12:14 PM
> On Jan 3, 2010, at 6:00 PM, Barney
> Cordoba wrote:
> 
> > 
> > 
> > --- On Sun, 1/3/10, Michael Tüxen <Michael.Tuexen at lurchi.franken.de>
> wrote:
> > 
> >> From: Michael Tüxen <Michael.Tuexen at lurchi.franken.de>
> >> Subject: Re: igb interrupt moderation
> >> To: "Mike Tancsa" <mike at sentex.net>
> >> Cc: "Barney Cordoba" <barney_cordoba at yahoo.com>,
> jfvogel at gmail.com,
> freebsd-net at freebsd.org
> >> Date: Sunday, January 3, 2010, 11:38 AM
> >> On Jan 3, 2010, at 5:23 PM, Mike
> >> Tancsa wrote:
> >> 
> >>> At 11:13 AM 1/3/2010, Michael Tüxen wrote:
> >>>>> 
> >>>>> Just a separate datapoint about this
> driver,
> >> unless I apply
> >>>>> 
> >>>>> http://people.freebsd.org/~yongari/igb/igb.buf.patch6
> >>>>> 
> >>>>> the driver is not really usable for me
> in
> >> RELENG_8 on the dual port version of the card
> >>>> Could you elaborate on what you mean by
> "not
> >> really usable"?
> >>> 
> >>> 
> >>> Hi,
> >>>         Some
> link state issues
> >> (getting confused about what port is up), problems
> at high
> >> packet rates.  I dont have this card in
> production, but
> >> in my test environment it was much more stable on
> RELENG_8
> >> with the above patch in that I was not able to
> wedge the
> >> box.  pps rates were pretty ok on a low end
> i7 as
> >> well.
> >> Thanks for the information. I'll give it a try. I
> have a
> >> problem when I flood
> >> a system with SCTP INITs. The system under attack
> becomes
> >> completely unresponsive
> >> on the console. However, it continues to send
> INIT-ACKs
> >> back. After the last
> >> commit from Jack it recovers after the attack. Not
> yet sure
> >> what is going on.
> >> Using the em driver does not have the problem.
> However,
> >> when using the em
> >> driver only one core is fully used, when using the
> igb
> >> driver both cores are fully
> >> used. Unfortunately I do not have a more than dual
> core
> >> machine available for
> >> this testing...
> > 
> > Try em and lower the interrupt moderation to something
> like 500 (about
> > 100 packets per int is good). The latency isn't going
> to be noticable and
> > you'll see your cpu burden reduced quite a bit. 
> I'll try. Thanks.
> > 
> > Are you using a single NIC on a server, or do you have
> a firewall or
> > bridge?
> The system is a sender/receiver for SCTP. I'm interested in
> the 82576
> since it provides checksum offloading for it. I use one or
> two ports
> for simultaneous data transfer. The cards using the em
> driver do
> not support this feature. So I'm trying to verify that the
> performance
> goes up when using hardware checksum. But under attack,
> this is currently
> not the case... 
> > 
> > Barney

Are you using just 1 queue? Just because you're using both cpus
doesn't mean its efficient. The  8257x has separate interrupts for 
transmit and receive, so 1 queue will be a closer match to the em
driver so you can gauge if the offload is effective. I don't know how
far jack has gotten in addressing the lock contention issue in igb.
Obviously, try all scenarios. What seems obvious rarely plays out in
practice.

Barney