em forwarding performance (was Proposed 6.2 em RELEASE
patch
Mike Tancsa
mike at sentex.net
Tue Nov 21 05:50:40 UTC 2006
I am moving this thread over to performance after this post as it
makes more sense to continue there.
At 11:54 PM 11/19/2006, Mike Tancsa wrote:
>At 04:30 PM 11/13/2006, Scott Long wrote:
>>Mike Tancsa wrote:
>>>At 12:15 AM 11/13/2006, Scott Long wrote:
>>>
>>>>Is this with EM_INTR_FAST enabled also?
>>>
>>>Without it, the 2 streams are definitely lossy on the management interface
>>>
>>> ---Mike
>>
>>Ok, and would you be able to test the polling options as well?
>
>
>Note about platforms. The HEAD w Patch is a patch
>glebius at freebsd.org asked me to test. FastFWD is with
>net.inet.ip.fastforwarding on. Also with FastFWD set to one, I
>always used the kernel options ADAPTIVE_GIANT commented out and
>added NO_ADAPTIVE_MUTEXES. INET6 was removed from all kernels as
>well. With these kernel changes, and fast forwarding on, I was able
>to keep the box r2 responsive from the console as while blasting
>packets across its 2 interfaces. Otherwise, the box seemingly
>livelocked. For the linux kernel config, it was pretty well the
>default, except I removed INET6, IPSEC and disabled iptables. The
>LINUX kernel was 2.6.18.2 on FC5.
>
>The first test is with UDP netperf.
>/usr/local/netperf/netperf -l 60 -H 192.168.44.1 -i 10,2 -I 99,10 -t
>UDP_STREAM -- -m 10 -s 32768 -S 32768
>/usr/local/netperf/netperf -l 60 -H 192.168.44.1 -i 10,2 -I 99,10 -t
>UDP_STREAM -- -m 64 -s 32768 -S 32768
>/usr/local/netperf/netperf -l 60 -H 192.168.44.1 -i 10,2 -I 99,10 -t
>UDP_STREAM -- -m 128 -s 32768 -S 32768
>/usr/local/netperf/netperf -l 60 -H 192.168.44.1 -i 10,2 -I 99,10 -t
>UDP_STREAM -- -m 200 -s 32768 -S 32768
Again, this is the same setup as described at http://www.tancsa.com/blast.jpg
My goals of this testing was to understand the following :
1) the new em driver to make sure it works well for me and give it a
good shake out under load for RELENG_6
2) understand the implications of various configs on forwarding
performance of SMP vs UP vs Polling vs Fast Interrupts and to avoid
livelock when there is a lot of pps
In this round of testing, I tried of RELENG_6 i386 in UP config as
well. Although raw packets / second (pps) forwarding was faster, the
box was pretty unresponsive and userland apps (i.e. routing) and made
the config pretty unusable with fast_forwarding enabled. Once ipfw
was loaded, the box would totally lock up and routing daemons was
start to spin out of control as hold timers expired. Bottom line,
there is slightly less raw pps performance out of an SMP config, but
the box seems to be far more resilient against a high pps attack.
RELENG_6 SMP with #define EM_FAST_INTR 1 in if_em.c
net.inet.ip.fastforwarding=1
with ADAPTIVE_GIANT removed from the kernel and
NO_ADAPTIVE_MUTEXES added
gives decent pps forwarding rates, without the box becoming live
locked. Note, in the real world, given an average packet size of
~600 bytes, a box in this config should be able to route and firewall
gigabit speeds without much issue and can sustain moderate DDoS pps
attacks over the net.
For the routing test, I used 2 peers each sending ~ 195K routes.
I re-tested the single UDP stream with 194k routes loaded in the
kernel routing table and 2 bgp peers. Then, while blasting across the
box, I cleared the session which had all the routes installed in the
kernel so that the daemon would have to reinstall all the routes to
point to the other peer. While this was happening (10 seconds on SMP
boxes, MUCH longer ~ 1min on UP, sometimes total failure) I was
measuring throughput. On UP it didnt drop too much, a bit more on
SMP, but convergence was quite fast, about 10 seconds. Similarly,
installing ipfw rules on the UP version made the box totally live
lock in non polling mode, but seemed to perform well enough in
polling mode. pf lagged quite far behind
The biggest difference seems to be in the efficiency of the firewall
rules in LINUX vs FreeBSD. Granted, the rules I inserted, are poorly
written, but adding rules did seem to have a linearly negative impact
on performance, where as rules via iptables did not significantly
impact forwarding rates. However, in the LINUX test, I seemed to
trigger some race in bgpd when doing the clear that took a little
more proding with FreeBSD, but is there as well :( So it might be
back to version .98
The table is also up at http://www.tancsa.com/blast.html which might
be easier to read
Straight Routing test One Stream 194K Routes bgp
clear and single ipfw 5 ipfw ruipfw 10 pf 1 pf 5
Linux 581,310 590,584
579,833 582,963 575,718 579,442
FreeBSD HEAD Nov 20
+FastFWD 540,695 529,425 439,980
398,283 370,458
FreeBSD HEAD Nov 11 441,560
RELENG6 i386 407,403
RELENG6 i386
FastFWD 557,580 562,547 484,250
425,791 386,644 353,856 333,293
FreeBSD HEAD w Patch 422,294
FreeBSD HEAD w Patch
FastFWD 567,290 564,340 482,093 436,205
381,459 359,248 361,458
AMD64 RELENG6 w
FastFWD 574,590 549,233 486,737
400,050 320,129 296,760 273,824
AMD64 RELENG6 polling 285,917
AMD64 RELENG6 polling FastFWD 512,042
RELENG6 i386 polling FastFWD 558,600 550,041
RELENG6 i386 polling FastFWD HZ=2000 565,520 563,068 373,904
RELENG_6 UP i386 400,188
RELENG_6 UP i386
FastFWD 584,900 582,534 560,033 560,323
RELENG_6 UP i386 FastFWD
Polling 585,934 476,629
422,633 393,301
Straight Routing test 2 streams opposite direction
Linux 473,810 452,205
408,932
FreeBSD HEAD Nov 11 204,043
FreeBSD HEAD nov 20 +
fastFWD 312,140 250,277
223,289 208,551
RELENG6 i386 165,461
RELENG6 i386
FastFWD 368,960 353,437
216,597 206,077 194,47669,50067,385
FreeBSD HEAD w Patch 127,832
FreeBSD HEAD w Patch
FastFWD 346,220 404,785
249,617 234,047157,603
AMD64 RELENG6 w Polling 155,659
AMD64 RELENG6 w Polling FastFWD 231,541
RELENG6 i386 polling FastFWD 319,350 312,621
RELENG6 i386 polling FastFWD HZ=2000 300,280 299,018
RELENG_6 UP i386 FastFWD
Polling 342,551
229,652 205,322
More information about the freebsd-performance
mailing list