em forwarding performance (was Proposed 6.2 em RELEASE patch

Mike Tancsa mike at sentex.net
Tue Nov 21 05:50:40 UTC 2006

I am moving this thread over to performance after this post as it 
makes more sense to continue there.

At 11:54 PM 11/19/2006, Mike Tancsa wrote:
>At 04:30 PM 11/13/2006, Scott Long wrote:
>>Mike Tancsa wrote:
>>>At 12:15 AM 11/13/2006, Scott Long wrote:
>>>>Is this with EM_INTR_FAST enabled also?
>>>Without it, the 2 streams are definitely lossy on the management interface
>>>         ---Mike
>>Ok, and would you be able to test the polling options as well?
>Note about platforms.  The HEAD w Patch is a patch 
>glebius at freebsd.org asked me to test.  FastFWD is with 
>net.inet.ip.fastforwarding on. Also with FastFWD set to one, I 
>always used the kernel options ADAPTIVE_GIANT commented out and 
>added NO_ADAPTIVE_MUTEXES.  INET6 was removed from all kernels as 
>well.  With these kernel changes, and fast forwarding on, I was able 
>to keep the box r2 responsive from the console as while blasting 
>packets across its 2 interfaces.  Otherwise, the box seemingly 
>livelocked.  For the linux kernel config, it was pretty well the 
>default, except I removed INET6, IPSEC and disabled iptables. The 
>LINUX kernel was on FC5.
>The first test is with UDP netperf.
>/usr/local/netperf/netperf -l 60 -H -i 10,2 -I 99,10 -t 
>UDP_STREAM -- -m 10 -s 32768 -S 32768
>/usr/local/netperf/netperf -l 60 -H -i 10,2 -I 99,10 -t 
>UDP_STREAM -- -m 64 -s 32768 -S 32768
>/usr/local/netperf/netperf -l 60 -H -i 10,2 -I 99,10 -t 
>UDP_STREAM -- -m 128 -s 32768 -S 32768
>/usr/local/netperf/netperf -l 60 -H -i 10,2 -I 99,10 -t 
>UDP_STREAM -- -m 200 -s 32768 -S 32768

Again, this is the same setup as described at http://www.tancsa.com/blast.jpg

My goals of this testing was to understand the following :

1) the new em driver to make sure it works well for me and give it a 
good shake out under load for RELENG_6

2) understand the implications of various configs on forwarding 
performance of SMP vs UP vs Polling vs Fast Interrupts and to avoid 
livelock when there is a lot of pps

In this round of testing, I tried of RELENG_6 i386 in UP config as 
well.  Although raw packets / second (pps) forwarding was faster, the 
box was pretty unresponsive and userland apps (i.e. routing) and made 
the config pretty unusable with fast_forwarding enabled.  Once ipfw 
was loaded, the box would totally lock up and routing daemons was 
start to spin out of control as hold timers expired.  Bottom line, 
there is slightly less raw pps performance out of an SMP config, but 
the box seems to be far more resilient against a high pps attack.

RELENG_6 SMP with #define EM_FAST_INTR 1  in if_em.c
with  ADAPTIVE_GIANT    removed from the kernel and

gives decent pps forwarding rates, without the box becoming live 
locked.  Note, in the real world, given an average packet size of 
~600 bytes, a box in this config should be able to route and firewall 
gigabit speeds without much issue and can sustain moderate DDoS pps 
attacks over the net.

For the routing test, I used 2 peers each sending ~ 195K routes.

I re-tested the single UDP stream with 194k routes loaded in the 
kernel routing table and 2 bgp peers. Then, while blasting across the 
box, I cleared the session which had all the routes installed in the 
kernel so that the daemon would have to reinstall all the routes to 
point to the other peer. While this was happening (10 seconds on SMP 
boxes, MUCH longer ~ 1min on UP, sometimes total failure) I was 
measuring throughput.  On UP it didnt drop too much, a bit more on 
SMP, but convergence was quite fast, about 10 seconds.  Similarly, 
installing ipfw rules on the UP version made the box totally live 
lock in non polling mode, but seemed to perform well enough in 
polling mode.  pf lagged quite far behind

The biggest difference seems to be in the efficiency of the firewall 
rules in LINUX vs FreeBSD.  Granted, the rules I inserted, are poorly 
written, but adding rules did seem to have a linearly negative impact 
on performance, where as rules via iptables did not significantly 
impact forwarding rates.  However, in the LINUX test, I seemed to 
trigger some race in bgpd when doing the clear  that took a little 
more proding with FreeBSD, but is there as well :(  So it might be 
back to version .98

The table is also up at http://www.tancsa.com/blast.html which might 
be easier to read

Straight Routing test One Stream                194K Routes  bgp 
clear and  single  ipfw 5 ipfw ruipfw 10    pf 1  pf 5
Linux                                     581,310   590,584 
579,833       582,963  575,718    579,442
FreeBSD HEAD Nov 20 
+FastFWD              540,695                   529,425       439,980 
  398,283    370,458
FreeBSD HEAD Nov 11                       441,560
RELENG6 i386                              407,403
RELENG6 i386 
FastFWD                      557,580   562,547         484,250 
425,791  386,644    353,856 333,293
FreeBSD HEAD w Patch                      422,294
FreeBSD HEAD w Patch 
FastFWD              567,290   564,340         482,093       436,205 
381,459    359,248 361,458
FastFWD                   574,590   549,233         486,737 
400,050  320,129    296,760 273,824
AMD64 RELENG6 polling                     285,917
AMD64 RELENG6 polling FastFWD             512,042
RELENG6 i386 polling FastFWD              558,600   550,041
RELENG6 i386 polling FastFWD HZ=2000      565,520   563,068         373,904
RELENG_6 UP i386                          400,188
RELENG_6 UP i386 
FastFWD                  584,900   582,534         560,033       560,323
RELENG_6 UP i386 FastFWD 
Polling          585,934                                 476,629 
422,633    393,301

Straight Routing test 2 streams opposite direction
Linux                                     473,810   452,205 
FreeBSD HEAD Nov 11                       204,043
FreeBSD HEAD nov 20 + 
fastFWD             312,140                                 250,277 
223,289    208,551
RELENG6 i386                              165,461
RELENG6 i386 
FastFWD                      368,960   353,437 
216,597  206,077    194,47669,50067,385
FreeBSD HEAD w Patch                      127,832
FreeBSD HEAD w Patch 
FastFWD              346,220   404,785 
249,617    234,047157,603
AMD64 RELENG6 w Polling                   155,659
AMD64 RELENG6 w Polling FastFWD           231,541
RELENG6 i386 polling FastFWD              319,350   312,621
RELENG6 i386 polling FastFWD HZ=2000      300,280   299,018
RELENG_6 UP i386 FastFWD 
Polling          342,551 
229,652    205,322

More information about the freebsd-performance mailing list