[POLLING] strange interrupt/system load

[Barney Cordoba]

--- On Mon, 9/14/09, rihad <rihad at mail.ru> wrote:

> From: rihad <rihad at mail.ru>
> Subject: Re: [POLLING] strange interrupt/system load
> To: "Bruce Evans" <brde at optusnet.com.au>
> Cc: freebsd-net at FreeBSD.org
> Date: Monday, September 14, 2009, 3:38 AM
> Bruce Evans wrote:
> > On Sat, 12 Sep 2009, rihad wrote:
> > 
> >> The box experiences ~230 mbit/s traffic flow
> through it. I've doubled some sysctls after reading
> polling(4):
> >> kern.polling.each_burst=10 # was: 5
> >> kern.polling.burst_max=350 # was: 150
> >> 
> >> FreeBSD 7.2-RELEASE-p3 amd64
> >> HZ=1000
> > 
> > How much better does it work without POLLING?
> > 
> Without polling (current load around 190-200 mbit/s, around
> 24-26 kpps):
> 
> top:
> CPU:  0.0% user,  0.0% nice,  8.4%
> system,  0.0% interrupt, 91.6% idle
> 
> Interrupts/s: 18322 total
> 28 mpt0 irq16
> 1999 cpu0: time
> 6906 em0 irq256
> 3392 em1 irq257
> 1999 cpu1: time
> 1999 cpu2: time
> 1999 cpu3: time

You really need to look at the taskq usage as averaging on a 4 core
system muddies things up. em will generally run on 1 core per NIC, and
interrupts are filtered so you won't see any interrupt usage. On a 
4 core system you could exhaust a core and still be at 25% overall, so 
you need to watch the max usage per core.

Things aren't measured properly in polling mode so its difficult to
compare them one to one. You really don't need to; intuitively it makes
zero sense to use polling with em.

You'll do a lot better setting your ITR to 2000 or so. You really don't
need an interrupt every 4 packets at those traffic levels.

Barney