Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

Kip Macy kip.macy at
Sun Jul 20 02:28:06 UTC 2008

On Sat, Jul 19, 2008 at 7:17 PM, Brian McGinty <brian.mcginty at> wrote:
> G'day Kip,
>> I'm hoping to get to it some time in August. I'm a bit behind in my
>> contracts at the moment.
> A few weeks ago, I did a quick comparison of the driver between
> FreeBSD and Linux, and found quite a few differences that's worth
> pulling over.  The guy from Intel working on FreeBSD, Jack?, is he the
> one that does this sort of sync-up of the drivers between the two
> distribution, or you?  There's been a lot of changes recently,
> including full support for multiple Rx/Tx queues that significantly
> ups the ante on performance. FreeBSD doesn't support multiple Rx/Tx,
> or does something half arsed.

This is on a variant of RELENG_6 FreeBSD with a recent version of ULE
and running the Checkpoint firewall. It also uses the full number of
queues available to igb (4) and #queues == #cores (8 in this case) for
ixgbe. The drivers in CVS have some bugs that I have fixed in this
FreeBSD variant. FreeBSD's CVS version of the Intel drivers definitely
lags Linux in terms of some optimizations. Even my version doesn't
have some of the linux optimizations.

>> FYI: I'm actually able to forward 2.3Mpps between 2 10Gig interfaces
>> on an 8-core system. I'm hoping to push it up to 3Mpps.

This is testing with an IXIA I don't currently have zero loss numbers.
This is not fully loaded. However, ixgbe spews out pause frames when
rx gets backed up so losses never get much above 0.1%.

> Is this no-loss number, and how did you test it?  I don't have
> throughput numbers for the Oplin.  I'm waiting to get some time on the
> Ixia at work to generate performance numbers for 1G and 10G for all
> packet sizes, on FreeBSD and Linux, on a 16 core system, and blast it
> to the list. I expect Linux to do 2-3 times better :-)

Sure, if you don't care about packet reordering. On their own box
Checkpoint claims that Linux is currently able to do 20% better than
we are seeing. Even they don't claim 200% - 300%. I know people who
are switching off of Linux for memcache because they simply can't make
it perform. So you're mileage really varies depending on the workload.
I'm not sure where you get your numbers from. I would really like to
get a hold of this magical Linux distribution to do a side by side
comparison on the same workload. A 200% - 300% performance delta would
definitely justify switching.


More information about the freebsd-net mailing list