[Bug 233759] igb (I210) + net.inet.ipsec.async_crypto=1 + aesni kill receiving queues and traffic

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233759

--- Comment #6 from Lev A. Serebryakov <lev at FreeBSD.org> ---
(In reply to Sean Bruno from comment #5)

I have three systems (they are separate physical systems, not VMs).

(1) Manager.
(2) Device Under Test ("DUT")
(3) Mirror.

Each system has 3 interfaces. One interface of each system is management one to
connect from outside work, and these interfaces is not in scope of this
description.

Manager system has two interfaces in question: "outbound" and "inbound".
  - outbound has IP 10.1.0.2/24 and it is connected with "inbound" interface of
DUT (via dedicated switch).
  - inbound has IP 10.10.10.2/24 and it is connected with "outbound"  interface
of "Mirror".
Manager system doesn't have any special routing record.

DUT system has two interfaces: "outbound" (igb1 in this ticket) and "inbound"
(igb0 in this ticket).
  - "outbound" (igb1) has IP 10.2.0.1/24 and it is connected with "inbound"
interface of "Mirror".
  - "inbound" (igb0) has IP 10.1.0.1/24 and it is connected with "outbound"
interface of "Manager" (via dedicated switch).
DUT has routing enabled and has "route -net 10.10.10.0/24 10.2.0.1".
DUT has such IPSec settings:
============
add 10.2.0.1 10.2.0.2 esp 0x10001 -m tunnel -E aes-gcm-16
"wxyz0123456789abcdef";
add 10.2.0.1 10.2.0.` esp 0x10002 -m tunnel -E aes-gcm-16
"wxyz0123456789abcdef";
spdadd 10.1.0.0/24 10.10.10.0/24 udp -P out ipsec
esp/tunnel/10.2.0.1-10.2.0.2/require;
spdadd 10.10.10.0/24 10.1.0.0/24 udp -P in  ipsec
esp/tunnel/10.2.0.2-10.2.0.1/require;
============

Mirror system has two interfaces in question: "outbound" and "inbound".
  - outbound has IP 10.10.10.1/24 and it is connected with "inbound" interface
of Manager.
  - inbound has IP 10.2.0.2/24 and it is connected with "outbound"  interface
of DUT.
Mirror has routing enabled and has "route -net 10.1.0.0/24 10.2.0.2".
Mirror has static ARP for 10.10.10.2-10.10.10.254 points to "Manager" "Inbound"
interface.
Mirror has such IPSec settings:
============
add 10.2.0.1 10.2.0.2 esp 0x10001 -m tunnel -E aes-gcm-16
"wxyz0123456789abcdef";
add 10.2.0.1 10.2.0.` esp 0x10002 -m tunnel -E aes-gcm-16
"wxyz0123456789abcdef";
spdadd 10.10.10.0/24 10.1.0.0/24 udp -P out ipsec
esp/tunnel/10.2.0.2-10.2.0.1/require;
spdadd 10.1.0.0/24 10.10.10.0/24 udp -P in  ipsec
esp/tunnel/10.2.0.1-10.2.0.2/require;
============

Ok, it is config. Really, it is loop "Manager -> DUT -> Mirror -> Manager"
where connection between DUT and Mirror has additional IPsec config. Manager
and Mirror are much more powerful than DUT and could pass full-wire-speed
traffic without any problems with and without encryption.

Now to test.

Manager generates (with netmap's pkt-gen) UDP traffic with such
characteristics:

Transmit interface: "outbound"
Dst MAC: DUT "inbound"
Src IPs: 10.1.0.2:2000-10.1.0.5:2004
Dst IPs: 10.10.10.2:2000-10.10.10.128:2006

Manager receives all traffic (with netmap's pkt-gen) at "inbound" interface and
measure bandwidth.

Now, if DUT has default setting for async IPsec (turned off) it could pass
690Mbit/s or 199Kp/s. Any traffic lower than that passes without any losses.
For example, if I generate traffic and speed 64P/s (without any prefixes!) I
see each and any packet returned to Manager from Mirror via DUT. No problems
here.


If I turn on async IPsec ("sysctl net.inet.ipsec.async_crypto=1" on DUT), no
matter which traffic is generated (I've tested with 64 packets per second, not
kilo-packets, simple packets!) receive queues of DUT inbound interface (igb0)
stop to work one by one.

-- 
You are receiving this mail because:
You are the assignee for the bug.