Linux SMP network performance measurements

[Ian FREISLICH]

Stefan Lambrev wrote:
> Thierry Herbelot wrote:
> > gives some measurements on various tweakings of an SMP machine with
> > 4 Xeon processors (it *shows* a nice improvement when using more
> > CPUs and more bonded Ethernet interfaces).
> >
> > Has some the machine (and the time, obviously) to make some of the
> > same measurements with the latest FreeBSD versions ?
>
> I'm planning to test network performance on FreeBSD + bridged
> interfaces, very soon, but my test servers are not so powerful as
> the server from this page :) Best that I'll have is 1x quad core
> processor, 4 port gigabit intel network card and 2GB RAM.

I did some testing about a year or two ago and with a recent current
of about 2 months ago.  I found that generally SMP was a performance
regression for the workload I tested - forwarding and filtering.
The single most significant contributer to network performance is
level 1 cache size.

I believe that the extreme cost of mutex acquisition on Intel cpus
is the main culprit for SMP network performance regression.  Coupled
with miniscule L1 cache size on the entire Intel CPU product line
gives pretty poor network performance.

Forwarding (routing between multiple interfaces) and filtering
(ipfw) IIRC with quad Intel e1000 NIC:

Dual Intel Xeon 2.8GHz:		240Kpps  12k L1 cache
Single Intel Xeon 2.8GHz:	380Kpps	 12k L1 cache
Core 2 Duo 1.8Ghz:		420kpps	 12k L1 cache
Single Pentium-M 1.8GHz:	550Kpps	 32k L1 cache
Dual AMD opteron 2GHz:		890Kpps	 64k L1 cache
Single AMD opteron 2GHz:	970Kpps	 64k L1 cache

All these hosts had 255 vlan interfaces with about 3000 routes and
about 30000 firewall rules, with a good spread of packets between
the interfaces with polling and fastforwarding.  I struggled to
generate enough packets to load the AMD routers.

I was interested in SMP due to additional processing for netflow
accounting and packet rate monitoring for DDoS detection and
mitigation.

I recomend to anyone using FreeBSD as a router or for any serious
workload to just plain forget about using Intel CPUs.

Ian

--
Ian Freislich