[Bug 280037] KTLS with Intel QAT may trigger kernel panics

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280037

--- Comment #22 from ss3bsd <3226388001@jcom.home.ne.jp> ---
Created attachment 272912
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=272912&action=edit
draft patch [B]

For reference, I have been using QAT stably since then under the following
conditions:
  - with a slightly refined patch [A]
  - kern.ipc.tls.cbc_enable: 0
  - do not disable or unload qat after its activation
  - used for KTLS, OpenSSL qatprovider (TLS handshake), IPsec, and
(infrequently) GELI
  - upgraded to 15.1 from 14.x when 15.1 was released

Note:
  - Suspending the system is outside the scope of my use case.
  - The patch uploaded in Comment 17 is not required for the stability if patch
[A] is applied. Instead, for performance-wise (off-topic), I am using another
patch [B] that launches a higher number of ktls worker threads than the number
of cpu cores.


Background (my observation) of the patches:
  The KTLS implementation (FreeBSD 13.x) used to use the QAT device
synchronously both for receive (RX) and send (TX) sides. It now (14.0 and
later) uses asynchronous dispatch for TX (RX is kept synchronous). 

  [A] The asynchronous dispatch improved performance but seems to have
introduced some instability such that a ktls session hangs or memory leaks if
the queue for asynchronous dispatch is full or (IIRC) a kernel panic occurs if
a ktls session is terminated (freed) while a command remains in the queue. This
patch is intended to prevent those issues (and introduce some sysctl parameters
that I used to switch software/hardware engine or synchronous/asynchronous
dispatch for analysis).

  [B] While the ktls implementation synchronously works with the engine for RX,
the number of ktls worker thread is limited to the number of cpu cores. This
may not be ideal if a hardware offload engine (e.g. qat) is used. If the number
of cpu cores is fewer, not enough number of commands are parallelly submitted,
then the engine is not fully utilized. This patch introduces a sysctl parameter
kern.ipc.tls.thread_overcommit by which 4 times as many ktls worker threads can
be created. (In my environment, the patch improved the benchmark (parallel ktls
wget) result of receive throughput from 5-6 Gbps to almost 10 Gbps for 10GbE.)

-- 
You are receiving this mail because:
You are the assignee for the bug.