state of packet forwarding in FreeBSD?

[John Jasen]

Our goal was to test whether or not FreeBSD currently is viable, as the
operating system platform for high speed routers and firewalls, in the
40 to 100 GbE range.

In our investigations, we tested 10.3, 11.0/-STABLE, -CURRENT, and a USB
stick from BSDRP using the FreeBSD routing improvements project
enhancements (https://wiki.freebsd.org/ProjectsRoutingProposal).

We've tried stock and netmap-fwd, have played around a little with
netmap itself and dpdk, with the results summarized below. The current
testing platform is a Dell PowerEdge R530 with a Chelsio T580-LP-CR dual
port 40GbE card.

Suggestions, examples for using netmap, etc, all warmly welcomed.

Further questions cheerfully answered to the best of our abilities.

a) On the positive side, it appears that 11.0 is much faster than 10.0,
which we tested several years ago. With appropriate cpuset tuning, 5.5
mpps is achievable using modern hardware. Using slightly older hardware,
(such as a Dell R720 with v3 xeons), around 5.2-5.3 mpps can be obtained.

b) On the negative side, between the various releases, netmap appeared
to be unstable with the Chelsio cards -- sometimes supported, sometimes
broken. Also, we're still trying to figure out netmap utilities, such as
vale-ctl and bridge, so any advice would be appreciated.

b.1) netmap-fwd is admittedly single-threaded and does not support IPv6.
These clearly showed in our tests, as we were unable to achieve over 2.5
mpps, saturating a single CPU and letting the others fall asleep.
However, bumping a single CPU queue from around 0.6 mpps to 2.5 mpps is
nothing to ignore, so it could be useful in some cases.

c) The routing improvement project USB stick performed incredibly,
achieving 8.5 mpps out of the box. However, it appears
(https://wiki.freebsd.org/ProjectsRoutingProposal/ConversionStatus),
that many of the changes are still pending review, and that things have
not moved much in the last 18 months
(https://svnweb.freebsd.org/base/projects/routing/)

d) We've not figured out dpdk  (dpdk.org) yet. Our first foray into the
test examples, and we're stuck trying to get the interfaces online.

-- John Jasen (jjasen at gmail.com)