Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

[Bruce Evans]

On Fri, 4 Jul 2008, Paul wrote:

> Numbers are maximum with near 100% cpu usage and some errors occuring, just 
> for testing.
> FreeBSD  7.0-STABLE FreeBSD 7.0-STABLE #6: Thu Jul  3 19:32:38 CDT 2008 
> root at foo:/usr/obj/usr/src/sys/ROUTER  amd64
> CPU: Dual-Core AMD Opteron(tm) Processor 2222 (3015.47-MHz K8-class CPU)
> NON-SMP KERNEL  em driver, intel 82571EB NICs
> fastforwarding on, isr.direct on, ULE, Preemption (NOTE: Interesting thing, 
> without preemption gets errors similar to polling)

PREEMPTION is certainly needed with UP.  Without it, interrupts don't
actually work (to work, they need to preempt the running thread, but
they often (usually?) don't do that).  Then with UP, there is a good
chance that the interrupt thread doesn't get scheduled to run for a
long time, but with SMP (especially with lots of CPUs) there is a good
chance that another CPU gets scheduled to run the interrupt thread.
em (unless misconfigured) doesn't have an interrupt thread; it uses a
taskq which might take even longer to be scheduled than an interrupt
thread.  I use PREEMPTION with UP and !PREEMPTION with SMP.

With polling, missed polls cause the same packet loss as not preempting.

> I tried polling, and I tried the polling patch that was posted to the list 
> and both work but generate too many errors (missed packets).
> Without polling the packet errors ONLY occur when the cpu is near 100% usage

Polling should also only cause packet loss when the CPU is near 100% usage,
but now transients of near 100% usually cause packet loss, while with
interrupts it takes a transient of > 100% on the competing interrupt-
driven resources to cause packet loss.

Pleas trim quotes.

Bruce