IPsec performace - netisr hits %100

[Mark Johnston]

On Sat, May 01, 2021 at 04:30:59PM +0300, Özkan KIRIK wrote:
> This bug is related to CCR. @Navdeep Parhar <np at freebsd.org> , @John Baldwin
> <jhb at freebsd.org> if you are interested to fix this bug related with CCR, I
> can test if you provide patches. Test environment is explained in my first
> email on this thread.
> 
> @Mark Johnston <markj at freebsd.org> Now again on stable/13,
> - with aesni, without netipsec/ipsec_input.c patch - 1.44Gbps - single
> netisr thread eats %100 cpu
> - with qat, without netipsec/ipsec_input.c patch - 1.88Gbps - single netisr
> thread eats %100 cpu
> - with aesni, with netipsec/ipsec_input.c patch - 1.33Gbps
> - with qat, with netipsec/ipsec_input.c patch - 2.85Gbps -
> 
> stable/13 results are better then stable/12 but not enough fast. There is
> something makes bottleneck for IPsec.

So with these results it looks like we have 4 crypto threads running,
which is what I'd expect for two pairs of IP addresses.  There is still
a single-threaded bottleneck.  I would suggest generating a flame graph
using DTrace and https://github.com/brendangregg/FlameGraph to see where
we're spending CPU time.  It would also be useful to know if we're
getting errors or drops anywhere.  The QAT (sysctl dev.qat.*.stats) and
ESP/AH (netstat -s -p (esp|ah)) counters would be a useful start, in
addition to counters from cxgbe.