40Gbps http client benchmark

[Eugene Grosbein]

Hi!

Is there any high performance benchmark acting as http client for outer http server
capable to receive 40Gbps without overwhelming CPU with insane number of syscalls?

I've tried benchmarks/wrk version 4.0.2 and it works just fine upto 20Gbps
for my hardware: two 6-core (HT disabled) Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
with two dual-port ix(4) 82599ES 10-Gigabit SFI/SFP+ Network Connection
combined to single lagg interface (lagghash l4).

But each worker pthread of wrk generates too many kqueue() system calls
polling for incoming data and eats 100% of its CPU core and cannot receive more.
Or, it may be some kqueue() kernel level lock contention, I do not know.
More worker threads, more overloaded CPU cores, no increase of transfer over about 20Gbps.

I transfer 1MByte-sized files for the benchmark (tried 8MB-sized too).
I set sysctl net.inet.tcp.recvspace=4194304 (4MB).
I've even patched wrk to to use read buffer sized 4MB instead of its compiled-in
default 8k, as it does not change default socket options other than TCP_NODELAY:

--- src/wrk.h.orig      2016-08-06 23:20:16.205906000 +0700
+++ src/wrk.h   2016-08-06 23:20:20.460579000 +0700
@@ -18,7 +18,7 @@
  #include "ae.h"
  #include "http_parser.h"

-#define RECVBUF  8192
+#define RECVBUF  4194304

  #define MAX_THREAD_RATE_S   10000000
  #define SOCKET_TIMEOUT_MS   2000

Nothing helps to decrease consumed system time and ktrace(1) assures
that's because of kevent() calls.