IPsec performace - netisr hits %100

[Özkan KIRIK]

the previous flamegraph is captured while iperf client (jail) sends to
iperf server.
the attached flamegraph to this mail is captured while iperf configured
full-duplex mode. Throughput is about up: 1.5 Gbps down: 1.5 Gbps total: 3
Gbps


On Sat, May 1, 2021 at 11:57 PM Özkan KIRIK <ozkan.kirik at gmail.com> wrote:

> Hello,
> The flamegraph is attached.
>
> # netstat -s
> ...
> ipsec:
>         0 inbound packets violated process security policy
>         0 inbound packets failed due to insufficient memory
>         0 invalid inbound packets
>         0 outbound packets violated process security policy
>         0 outbound packets with no SA available
>         0 outbound packets failed due to insufficient memory
>         0 outbound packets with no route available
>         0 invalid outbound packets
>         0 outbound packets with bundled SAs
>         0 spd cache hits
>         0 spd cache misses
>         0 clusters copied during clone
>         0 mbufs inserted during makespace
> ah:
>         0 packets shorter than header shows
>         0 packets dropped; protocol family not supported
>         0 packets dropped; no TDB
>         0 packets dropped; bad KCR
>         0 packets dropped; queue full
>         0 packets dropped; no transform
>         0 replay counter wraps
>         0 packets dropped; bad authentication detected
>         0 packets dropped; bad authentication length
>         0 possible replay packets detected
>         0 packets in
>         0 packets out
>         0 packets dropped; invalid TDB
>         0 bytes in
>         0 bytes out
>         0 packets dropped; larger than IP_MAXPACKET
>         0 packets blocked due to policy
>         0 crypto processing failures
>         0 tunnel sanity check failures
>         AH output histogram:
>                 aes-gmac-128: 35517864
> esp:
>         0 packets shorter than header shows
>         0 packets dropped; protocol family not supported
>         0 packets dropped; no TDB
>         0 packets dropped; bad KCR
>         0 packets dropped; queue full
>         20 packets dropped; no transform
>         0 packets dropped; bad ilen
>         0 replay counter wraps
>         0 packets dropped; bad encryption detected
>         0 packets dropped; bad authentication detected
>         0 possible replay packets detected
>         23598941 packets in
>         11918943 packets out
>         0 packets dropped; invalid TDB
>         32247932688 bytes in
>         630318292 bytes out
>         0 packets dropped; larger than IP_MAXPACKET
>         0 packets blocked due to policy
>         0 crypto processing failures
>         0 tunnel sanity check failures
>         ESP output histogram:
>                 aes-gcm-16: 35517864
>
> dev.qat.1.stats.sym_alloc_failures: 0
> dev.qat.1.stats.ring_full: 1267
> dev.qat.1.stats.gcm_aad_updates: 0
> dev.qat.1.stats.gcm_aad_restarts: 0
> dev.qat.1.%domain: 0
> dev.qat.1.%parent: pci16
> dev.qat.1.%pnpinfo: vendor=0x8086 device=0x37c8 subvendor=0x8086
> subdevice=0x0000 class=0x0b4000
> dev.qat.1.%location: slot=0 function=0 dbsf=pci0:182:0:0
> dev.qat.1.%driver: qat
> dev.qat.1.%desc: Intel C620/Xeon D-2100 QuickAssist PF
> dev.qat.0.stats.sym_alloc_failures: 0
> dev.qat.0.stats.ring_full: 0
> dev.qat.0.stats.gcm_aad_updates: 0
> dev.qat.0.stats.gcm_aad_restarts: 0
> dev.qat.0.%domain: 0
> dev.qat.0.%parent: pci15
> dev.qat.0.%pnpinfo: vendor=0x8086 device=0x37c8 subvendor=0x8086
> subdevice=0x0000 class=0x0b4000
> dev.qat.0.%location: slot=0 function=0 dbsf=pci0:181:0:0
> dev.qat.0.%driver: qat
> dev.qat.0.%desc: Intel C620/Xeon D-2100 QuickAssist PF
> dev.qat.%parent:
>
>
>
>
> On Sat, May 1, 2021 at 5:51 PM Mark Johnston <markj at freebsd.org> wrote:
>
>> On Sat, May 01, 2021 at 04:30:59PM +0300, Özkan KIRIK wrote:
>> > This bug is related to CCR. @Navdeep Parhar <np at freebsd.org> , @John
>> Baldwin
>> > <jhb at freebsd.org> if you are interested to fix this bug related with
>> CCR, I
>> > can test if you provide patches. Test environment is explained in my
>> first
>> > email on this thread.
>> >
>> > @Mark Johnston <markj at freebsd.org> Now again on stable/13,
>> > - with aesni, without netipsec/ipsec_input.c patch - 1.44Gbps - single
>> > netisr thread eats %100 cpu
>> > - with qat, without netipsec/ipsec_input.c patch - 1.88Gbps - single
>> netisr
>> > thread eats %100 cpu
>> > - with aesni, with netipsec/ipsec_input.c patch - 1.33Gbps
>> > - with qat, with netipsec/ipsec_input.c patch - 2.85Gbps -
>> >
>> > stable/13 results are better then stable/12 but not enough fast. There
>> is
>> > something makes bottleneck for IPsec.
>>
>> So with these results it looks like we have 4 crypto threads running,
>> which is what I'd expect for two pairs of IP addresses.  There is still
>> a single-threaded bottleneck.  I would suggest generating a flame graph
>> using DTrace and https://github.com/brendangregg/FlameGraph to see where
>> we're spending CPU time.  It would also be useful to know if we're
>> getting errors or drops anywhere.  The QAT (sysctl dev.qat.*.stats) and
>> ESP/AH (netstat -s -p (esp|ah)) counters would be a useful start, in
>> addition to counters from cxgbe.
>>
>