Flamegraph of 8KB message bottlenecks

[Ryan Stone]

I brought up on the -transport call last week that I was having trouble
receiving 8KB packets at 40Gbps.  I blamed "TCP stack" peformance, although
what I really meant was "kernel performance" -- I just assumed that the
problem was in the stack.  From looking at a flamegraph, I'm not so sure
this was the case.  The flamegraph was taken on a "FreeBSD 10.1-ish"
system.  Unfortunately running stock FreeBSD head in this environment is
quite difficult, so I probably wouldn't be able to check if anything on
head has alleviated things.  You can see the flamegraph for youself here:

https://people.freebsd.org/~rstone/incast_8kb_flame.svg

You'll note that my underland application is way off in the bottom left,
taking 3% of all cycles.  Meanwhile top reports that the system is
completely bogged down in interrupt and system time.