Strange CPU distributionat very high level bandwidth

[Ivan Voras]

Invernizzi Fabrizio wrote:
> Hi all
> 
> i am going on with some performance tests on a 10gbe network card with FreeBSD.
> 
> I am doing this test: I send UDP traffic to be forwarded to the other port of the card on both the card ports.
> Using 1492-long packets i am uppering the number  of packets per second i sent In order to see wich is the maximum bandwidth (or pps) the system can support without losses.
> 
> The limit seems to be about 1890Mbps per port (3870 Mbps total).
> Looking more in deep the CPU behaviour i see this :
>   - uppering the sent pps results in uppering the intterrupt time (about 90%)
>   - when i am very strict to the limit, interrupt time falls to about 10% and CPU is always (85%) in system (rx/tx driver procedure)
> 
> Questions:
> - Is not the AIM intended to contrast this behaviour to limit interrupts sent to CPU? (nothing changes if i disable it)
> - Why does the system start loosing pkts in that condition?
> - Why does the system seem to perform better when it is managing more context switches?
> 

> - FreeBSD 7.2-RELEASE (64 bit)

One idea for you, not directly tied to forwarding as is but to the
recent development of multithreaded packet acceptance code, is to use
8.x (currently in BETA so usual precautions about debugging being
enabled apply) and then play with netisr and worker thread settings.

See the source here:

http://svn.freebsd.org/viewvc/base/head/sys/net/netisr.c?view=markup&pathrev=195078

and the comments starting at "Three direct dispatch policies are supported".

The code is experimental and thus disabled in 8.0 unless a combination
of the following loader tunables are set:

net.isr.direct_force
net.isr.direct
net.isr.maxthreads
net.isr.bindthreads

I think you can start simply by turning off net.isr.direct_force and
then start increasing net.isr.maxthreads until the benefits (if any) go
away. Since it is experimental code, your benchmarks would be nice to have.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 259 bytes
Desc: OpenPGP digital signature
Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20090820/63d1d302/signature.pgp