Should we enable KERN_TLS on amd64 for FreeBSD 13?

[Allan Jude]

On 2021-01-25 15:33, Andrew Gallatin wrote:
> On 1/25/21 2:59 PM, John Baldwin wrote:
>> On 1/25/21 10:45 AM, Allan Jude wrote:
>>> On 2021-01-08 12:26, Andrew Gallatin wrote:
>>>>
>>>> Kernel TLS (KTLS) support was added roughly a year ago, and provides
>>>> an efficient software or hardware accelerated path to have the kernel
>>>> (or the NIC) handle TLS crypto.  This is quite useful for web and
>>>> NFS servers, and provides a huge (2x -> 5x) efficiency gain by
>>>> avoiding data copies into userspace for crypto, and potentially
>>>> offloading the crypto to hardware.
>>>>
>>>>
>>>> KTLS is well tested on amd64, having been used in production at Netflix
>>>> for nearly 4 years.   The vast majority of Netflix video has been
>>>> served
>>>> via KTLS for the last few years.  Its what has allowed us to serve
>>>> 100Gb/s on Xeon 2697A cpus for years, and what allows us to serve
>>>> nearly 400Gb/s on AMD servers with NICs which support crypto offload.
>>>>
>>>> I have received a few requests to enable it by default in GENERIC, and
>>>> I'd like to get some opinions.
>>>>
>>>> There are essentially 3 options
>>>>
>>>> 1) Fully enable KTLS by adding 'options KERN_TLS' to GENERIC, and
>>>> flipping kern.ipc.tls.enable=1
>>>>
>>>> The advantage of this is that it "just works" out of the box for users,
>>>> and for reviewers.
>>>>
>>>> The drawback is that new code is thrust on unsuspecting users,
>>>> potentially exposing them to bugs that we have not found in our
>>>> somewhat limited web serving workload.
>>>>
>>>> 2) Enable KTLS in GENERIC, but leave it turned off by default.
>>>>
>>>> This option allows users to enable ktls without a rebuild of GENERIC,
>>>> but does not enable it by default. So they can enable it if they
>>>> know about it, but are protected from bugs.
>>>>
>>>> The disadvantages of this are that it increases the kernel size
>>>> by ~20K, starts up one thread per core on every amd64 machine,
>>>> and it adds more required tuning to get good performance from FreeBSD.
>>>>
>>>>
>>>> 3) Continue along with KTLS disabled in GENERIC
>>>>
>>>> This is the lowest risk, but adds a higher bar for users wanting
>>>> to use ktls.
>>>>
>>>>
>>>>
>>>> Note that the discussion is focused on amd64 only, as KTLS will
>>>> only work on 64-bit platforms which use a direct map.  It has
>>>> not been tested at all on ppc64, and currently causes a
>>>> panic-at-boot on arm64 due to what are suspected to be problems
>>>> in the arm64 PCB setup. See:
>>>> https://urldefense.com/v3/__https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247945__;!!OToaGQ!7pQUcHPbxA12vEdKTCp5jkyVxDCqYEJ-BI38kgHqGgweT7yYYG1BVhbDek0_Jc7mqA$
>>>>
>>>> Drew
>>>>
>>>
>>> Just before this went in, Ed cleaned up the arm64 GENERIC to get it
>>> closer to the amd64 one. Can we enable KERN_TLS in arm64 GENERIC as
>>> well?
>>
>> Well, I also fixed a bug KERN_TLS exposed on arm64 that was gating for
>> this (247945).  I would not be opposed to enabling it on arm64, but I
>> have not personally tested it on arm64.  If someone can verify it works
>> ok on arm64 I'd be happy for it to be enabled there.
>>
> 
> Yeah, that's the thing, I have much less confidence in ktls on arm64
> because we have not run it in production recently.  So I'm personally
> much less confident in enabling it on arm64.
> 
> Drew

Klara has tested it on arm64 fairly heavily, and only found an issue
with OpenSSL, but not found any issues with KERN_TLS itself.

-- 
Allan Jude