Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

[Robert Watson]

On Sat, 5 Jul 2008, Paul wrote:

> ULE + PREEMPTION for non SMP no major differences with SMP with ULE/4BSD and 
> preemption ON/OFF
>
> 32 bit UP test coming up with new cpu and I'm installing dragonfly sometime 
> this weekend :] UP: 1mpps in one direction with no firewall/no routing table 
> is not too bad, but 1mpps both directions is the goal here 700kpps with full 
> bgp table in one direction is not too bad Ipfw needs a lot of work, barely 
> gets 500kpps with no routing table with a few ipfw rules loaded.. that's 
> horrible Linux barely takes a hit when you start loading iptables rules , 
> but then again linux has a HUGE problem with routing random packet 
> sources/ports .. grr My problem Is I need some box to do fast routing and 
> some to do firewall.. :/ I'll have 32 bit 7-stable UP test with ipfw/routing 
> table and then move on to dragonfly. I'll post the dragonfly results here as 
> well as sign up for their mailing list.

First off, I would recommend using an 8-CURRENT kernel where possible 
(obviously, with all debugging features disabled), because that's where most 
of the work is going on right now.  MFCs are scheduled for quite a bit of it, 
but over the course of several months, so using the 8-CURRENT kernel would 
allow you to help us test and exercise the new code, as well as improve our 
confidence in it so that it can be MFC'd in a timely manner :-).

Experience suggests that forwarding workloads see significant lock contention 
in the routing and transmit queue code.  The former needs some kernel hacking 
to address in order to improve parallelism for routing lookups.  The latter is 
harder to address given the hardware you're using: modern 10gbps cards 
frequently offer multiple transmit queues that can be used independently 
(which our cxgb driver supports), but 1gbps cards generally don't.

LOCK_PROFILING is an excellent tool for diagnosing locking hot spots -- it has 
a significant performance hit, but the results are generally accurate despite 
this.  If your hardware supports hwpmc, that is also an excellent tool for 
monitoring what's going on.  Seeing snapshots of, say, 10-20 seconds of 
profiling in the steady state, would help us understand better what is going 
on in your environment.

There's some quite interesting work going on to improve network memory 
allocator efficiency, but that's a bit aways from commit to 8.x as I 
understand it, and probably not on the 7.x merge path due to the potential 
disruption it could cause.

There's also a patch going around to offload the em_start function to the task 
queue that processes its input, which significantly reduces lock contention if 
you have bursty transmit.  I'll see if I can't dig it up.

Robert N M Watson
Computer Laboratory
University of Cambridge

>
>
> Bart Van Kerckhove wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>> 
>> Paul / Ingo,
>> 
>>>> I tried all of this :/  still, 256/512 descriptors seem to work the
>>>> best. Happy to let you log into the machine and fiddle around if you
>>>> want :) 
>> I've been watching this thread closely, since I'm in a very similair
>> situation.
>> A few questions/remarks:
>> 
>> Does ULE provide better performance than 4BSD for forwarding?
>> Did you try freebsd4 as well? This thread had a report about that quite
>> opposite to my own experiences, -4 seemed to be a lot faster at forwarding
>> than anything else I 've tried so far.
>> Obviously the thing I'm interested in is IMIX - and 64byte packets.
>> Does anyone have any benchmarks for DragonFly? I asked around on IRC, but
>> that nor google turned up any useful results.
>> 
>> <snip> 
>>> I don't think you will be able to route 64byte packets at 1gbit
>>> wirespeed (2Mpps) with a current x86 platform.
>>> 
>> Are there actual hardware related reasons this should not be possible, or
>> is this purely lack of dedicated work towards this goal?
>> 
>> <snip>
>> 
>>> Theres a "sun" used at quagga dev as bgp-route-server.
>>> http://quagga.net/route-server.php
>>> (but they don't answered my question regarding fw-performance).
>>> 
>> 
>> 
>> the Quagga guys are running a sun T1000 (niagara 1) route server - I happen
>> to have the machine in my racks,
>> please let me know if you want to run some tests on it, I'm sure they won't
>> mind ;-)
>> It should also make a great testbed for SMP performance testing imho (and
>> they're pretty cheap these days)
>> Also, feel free to use me as a relay for your questions, they're not always
>> very reachable.
>> <snap>
>>
>> 
>>> Perhaps you have some better luck at some different hardware systems
>>> (ppc, mips, ..?) or use freebsd only for routing-table-updates and
>>> special network-cards (netfpga) for real routing.
>>> 
>> The netfpga site seems more or less dead - is this project still alive?
>> It does look like a very interesting idea, even though it's currently quite
>> linux-centric (and according to docs doesn't have VLAN nor ip6 support, the
>> former being quite a dealbreaker)
>> 
>> Paul: I'm looking forward to the C2D 32bit benchmarks (maybe throw in a
>> freebsd4 and/or dragonfly bench if you can..) - appreciate the lots of
>> information you are providing us :)
>> 
>> Met vriendelijke groet / With kind regards,
>> 
>> Bart Van Kerckhove
>> http://friet.net/pgp.txt
>> 
>> -----BEGIN PGP SIGNATURE-----
>> 
>> iQA/AwUBSG/tMgoIFchBM0BKEQKUSQCcCJqsw2wtUX7HQi050HEDYX3WPuMAnjmi
>> eca31f7WQ/oXq9tJ8TEDN3CA
>> =YGYq
>> -----END PGP SIGNATURE-----
>> 
>>
>> 
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>