[Bug 246951] carp(4): Active CARP member crashes: panic, trap_pfault, ip_input || ip_output when using ipSec, AES-NI (on Intel I350)

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246951

--- Comment #9 from freebsd-bugzilla at biscuit.ninja ---
(In reply to Kubilay Kocak from comment #4)

Thank you.

I've attached dmesg.boot output.

This is pfSense so there is no rc.conf. I've attached instead:
 - ifconfig output
 - interfaces and LAGGs specified in pfSense config.xml
 - sysctl tunables specified in pfSense config.xml

The uptime between crashes varies between 2 and 30 days. It does not seem to
correlate to any specific event that we are aware of or even peak throughput.
The only additional package installed on these firewalls is NRPE.

In terms of workload:
 - HTTP/s traffic too and from customers
 - TCP load blancing of customer HTTP/s with 10 pools, 4 virtual servers per
pool. Total of around 1.5 - 2 million active sessions
 - ipSec site-to-site tunnel for replication to our standby data centre
 - CARP / pfSync with bandwith/packet rates of 22-80 Mb/s, 2-8 Kpps
 - AES-NI enabled for IpSec (AES256-GCM)

The firewalls are handling:
 - 20-45 Mb/s (13-45 Kpps) inbound ipSec
 - 30-150 Mb/s (14-55 Kpps) outbound ipSec
 - 20-90 Mb/s (15-60 Kpps) inbound IP traffic
 - 50-250 Mb/s (15-60 Kpps) outbound IP traffic
 - 30-90k states
 - ~66k Mbuf Clusters utilised (out of 1M total)

The only other thing of note, that I can think of, is that we have a Cassandra
cluster replicating over the IpSec tunnel. That's around 256 constantly
changing states as data is replicated from one data centre to another.

We have now disabled IpSec and switched to OpenVPN for the site-to-site VPN, in
order to see whether the crash is reproducable without IpSec

Additionally, I had setup a couple of FreeBSD 11.3 VMs with a site-to-site
IpSec connection. I had continuous iperf running over the tunnel for 7 days
without issue.

If there is any further information that I can provide, or anything I can do to
assist, please don't hesitate.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.