netmap: recommended NIC for 40GbE capture on Linux?

[Jim Thompson]

> Before we spend money, I'd love to hear someone report success with capturing a single flow at >4Mpps, >20Gbps using netmap on Linux and
> what NIC they use.

You said linux, and this is freebsd-net, but this blog post (from yesterday) is probably apt.

https://blog.cloudflare.com/single-rx-queue-kernel-bypass-with-netmap/

> Ideally this would include transparent mode working so that ARP, ICMP etc can be handled by the host stack.

netmap-fwd (written by loos@ and announced yesterday at BSDcon Brazil) does this (re-injecting frames designed for ’this’ host on the host ring, though there is a mode
to handle ARP and ICMP in the fastpath code).

netmap-fwd is an implementation of the ip_fastforward code from FreeBSD on top of netmap.   It currently does around 5Mpps, single-stream, on a single core using
Chelsio or Intel NICs.  It’s IPv4-only right now, but once we add IPv6(*), it will be open sourced.

PPS @ 64-byte frames (pkt-gen) through DUT
Device under test                                 Network interface   Kernel forwarding   Fastforward enabled  Tryforward patch    netmap-fwd application
C2358 (2 core, 1.7 Ghz)                       Intel I354               123 kbps                 217 kbps                    213 kbps                 945 kpps
C2758 (8 core, 2.4 GHz)                      Chelsio T520 10G  270 apps               426 apps                    431 apps                  1.683 Mpps
Xeon-D 1540 (8 core, 2 GHz)               Intel X552 10G      439 kbps                557 kbps                    544 kbps                  2.230 Mpps
Xeon E3-1275 (4 core, 3.5 GHz)          Intel X520-2           1.058 Mpps           1.331 Mpps                1.297 Mpps              5.053 Mpps

‘Tryforward' above is the code in https://reviews.freebsd.org/D3737

(We haven’t tried it on linux.)

I find it likely that an application that acts as a simple filter like https://github.com/jibi/nm-single-rx-queue/blob/master/main.c or https://github.com/erikarn/netmap-tools/tree/master/src/bridge 
will greatly exceed the 5Mpps figure above.   See also: http://adrianchadd.blogspot.com.br/2015/04/intel-ddio-llc-cache-buffer-alignment.html

Jim
(*) this work will need to wait until after we can get pfSense 2.3 released