Reducing number of interrupts from intel pro 1000 et adapter

[Barney Cordoba]

--- On Wed, 12/2/09, Yuriy A. Korobko <administrator at shtorm.com> wrote:

> From: Yuriy A. Korobko <administrator at shtorm.com>
> Subject: Re: Reducing number of interrupts from intel pro 1000 et adapter
> To: "Barney Cordoba" <barney_cordoba at yahoo.com>
> Cc: freebsd-net at freebsd.org
> Date: Wednesday, December 2, 2009, 11:05 AM
> > > 
> > > I'd like to know a way to control tx interrupts
> on intel
> > > pro 1000 et
> > > adapter with igb driver. Just installed one in
> the router
> > > and systat
> > > shows 8-9k rx interrupts and 20k tx interrupts
> from igb0
> > > and igb1
> > > adapters. Box is a router running freebsd 7.2
> release, I've
> > > tried
> > > default driver from kernel source and latest from
> intel
> > > site, effect is
> > > the same with automatic interrupt moderation
> enabled and
> > > disabled. I
> > > have the same box with intel pro 1000 pt adapter
> which
> > > have
> > > tx(rx)_int_delay sysctls in em driver, I was able
> to reduce
> > > number of
> > > tx/rx interrupts to 7-8k per interface and got
> much more
> > > cpu idle
> > > because of less context switches with same pps.
> > > 
> 
> 
> 
> > I'm curious as to why you are doing a load test on a
> single core system
> > with a part that is clearly designed to be used on a
> multicore system?
> > 
> > Barney
> 
> Box have core quad cpu:
> 
> 87 processes:  7 running, 58 sleeping, 22 waiting
> CPU 0:  0.0% user,  0.0% nice,  3.1% system,
> 36.4% interrupt, 60.5% idle
> CPU 1:  0.0% user,  0.0% nice, 15.5% system,
> 41.1% interrupt, 43.4% idle
> CPU 2:  0.0% user,  0.0% nice, 11.6% system,
> 27.1% interrupt, 61.2% idle
> CPU 3:  0.0% user,  0.0% nice,  2.3% system,
> 41.5% interrupt, 56.2% idle
> Mem: 192M Active, 206M Inact, 246M Wired, 716K Cache, 112M
> Buf, 1356M 
> 
> Anyway, high interrupt rate fixed by
> 
> border# cat /boot/loader.conf | grep igb
> if_igb_load="YES"
> hw.igb.rxd=4096
> hw.igb.txd=4096
> hw.igb.num_queues=1
> hw.igb.enable_aim=1
> hw.igb.low_latency=1000
> hw.igb.ave_latency=2000
> hw.igb.bulk_latency=4000
> hw.igb.rx_process_limit=200
> hw.igb.fc_setting=0
> hw.igb.lro=0
> border# 
> 
> Have no idea why, but it seems like setting dev.igb.
> sysctls at runtime
> just ignored by the driver. Now I have 1000-1200 rx and
> around 2000 tx
> interrupts per second from interface with 100 kpps flow in
> each
> direction, main cpu eater on the router is dummynet right
> now, it will
> be awesome if one day we can have it multithreaded. 
> 
> I'm using latest igb driver from intel site, default driver
> from kernel
> may not have this issue.
> 
> Thanks.
> 
> 
> _______________________________________________
> freebsd-net at freebsd.org
> mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
> 

Ah, dummynet. Explains much. I've never seen livelock on a quad core
system.

The igb driver in FreeBSD needs a complete and total overhaul. Don't 
waste alot of time tweeking it. Turn off AIM and hard code something
that suits your network. Intel's aim algorithm is ill-conceived IMO.


Barney