Should we enable KERN_TLS on amd64 for FreeBSD 13?

[Rodney W. Grimes]

> 
> Kernel TLS (KTLS) support was added roughly a year ago, and provides
> an efficient software or hardware accelerated path to have the kernel
> (or the NIC) handle TLS crypto.  This is quite useful for web and
> NFS servers, and provides a huge (2x -> 5x) efficiency gain by
> avoiding data copies into userspace for crypto, and potentially
> offloading the crypto to hardware.
> 
> 
> KTLS is well tested on amd64, having been used in production at Netflix
> for nearly 4 years.   The vast majority of Netflix video has been served
> via KTLS for the last few years.  Its what has allowed us to serve
> 100Gb/s on Xeon 2697A cpus for years, and what allows us to serve
> nearly 400Gb/s on AMD servers with NICs which support crypto offload.
> 
> I have received a few requests to enable it by default in GENERIC, and
> I'd like to get some opinions.
> 
> There are essentially 3 options
> 
> 1) Fully enable KTLS by adding 'options KERN_TLS' to GENERIC, and
> flipping kern.ipc.tls.enable=1
> 
> The advantage of this is that it "just works" out of the box for users,
> and for reviewers.
> 
> The drawback is that new code is thrust on unsuspecting users,
> potentially exposing them to bugs that we have not found in our
> somewhat limited web serving workload.
> 
> 2) Enable KTLS in GENERIC, but leave it turned off by default.
> 
> This option allows users to enable ktls without a rebuild of GENERIC,
> but does not enable it by default. So they can enable it if they
> know about it, but are protected from bugs.
> 
> The disadvantages of this are that it increases the kernel size
> by ~20K, starts up one thread per core on every amd64 machine,
> and it adds more required tuning to get good performance from FreeBSD.
> 
> 
> 3) Continue along with KTLS disabled in GENERIC
> 
> This is the lowest risk, but adds a higher bar for users wanting
> to use ktls.

4) If possible could the few places that have #ifdef be dealt with
   using a switch table or IFUNCs so that the KTLS code could be
   enable at boot time via a loadable module?

> 
> Note that the discussion is focused on amd64 only, as KTLS will
> only work on 64-bit platforms which use a direct map.  It has
> not been tested at all on ppc64, and currently causes a
> panic-at-boot on arm64 due to what are suspected to be problems
> in the arm64 PCB setup. See:
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247945
> 
> Drew
> 
-- 
Rod Grimes                                                 rgrimes at freebsd.org