how to use the ktls

[John Baldwin]

On 1/26/20 8:08 PM, Rick Macklem wrote:
> John Baldwin wrote:
> [stuff snipped]
>> Hmmm, this might be a fair bit of work indeed.
>>
>> Right now KTLS only works for transmit (though I have some WIP for receive).
>>
>> KTLS does assumes that the initial handshake and key negotiation is handled by
>> OpenSSL.  OpenSSL uses custom setockopt() calls to tell the kernel which
>> session keys to use.
>>
>> I think what you would want to do is use something like OpenSSL_connect() in
>> userspace, and then check to see if KTLS "worked".  If it did, you can tell
>> the kernel it can write to the socket directly, otherwise you will have to
>> bounce data back out to userspace to run it through SSL_write() and have
>> userspace do SSL_read() and then feed data into the kernel.
>>
>> The pseudo-code might look something like:
>>
>> SSL *s;
>>
>> s = SSL_new(...);
>>
>> /* fd is the existing TCP socket */
>> SSL_set_fd(s, fd);
>> OpenSSL_connect(s);
>> if (BIO_get_ktls_send(SSL_get_wbio(s)) {
>>    /* Can use KTLS for transmit. */
>> }
>> if (BIO_get_ktls_recv(SSL_get_rbio(s)) {
>>    /* Can use KTLS for receive. */
>> }
> 
> So, I've been making some progress. The first stab at the daemons that do the
> handshake are now on svn in base/projects/nfs-over-tls/usr.sbin/rpctlscd and
> rpctlssd.
> 
> A couple of questions...
> 1 - I haven't found BIO_get_ktls_send() or BIO_get_ktls_recv(). Are they in some
>        different library?

They only existing currently in OpenSSL master (which will be OpenSSL 3.0.0 when it
is released).  I have some not-yet-tested WIP changes to backport those changes into
the base OpenSSL, but it will also add overhead to future OpenSSL imports perhaps,
so it is something I need to work with secteam@ on to decide if it's viable once I
have a tested PoC.

I will try to at least provide a patch to the security/openssl port to add a KTLS
option "soon" that you could use for testing.

> 2 - After a successful SSL_connect(), the receive queue for the socket has 478bytes
>       of stuff in it. SSL_read() seems to know how to skip over it, but I haven't
>       figured out a good way to do this. (I currently just do a recv(..478,0) on the
>       socket.)
>       Any idea what to do with this? (Or will the receive side of the ktls figure out
>       how to skip over it?)

I don't know yet. :-/  With the TOE-based TLS I had been testing with, this doesn't
happen because the NIC blocks the data until it gets the key and then it's always
available via KTLS.  With software-based KTLS for RX (which I'm going to start
working on soon), this won't be the case and you will potentially have some data
already ready by OpenSSL that needs to be drained from OpenSSL before you can
depend on KTLS.  It's probably only the first few messsages, but I will need to figure
out a way that you can tell how much pending data in userland you need to read via
SSL_read() and then pass back into the kernel before relying on KTLS (it would just
be a single chunk of data after SSL_connect you would have to do this for).

> I'm currently testing with a kernel that doesn't have options KERN_TLS and
> (so long as I get rid of the 478 bytes), it then just does unencrypted RPCs.
> 
> So, I guess the big question is.... can I get access to your WIP code for KTLS
> receive? (I have no idea if I can make progress on it, but I can't do a lot more
> before I have that.)

The WIP only works right now if you have a Chelsio T6 NIC as it uses the T6's TCP
offload engine to do TLS.  If you don't have that gear, ping me off-list.  It
would also let you not worry about the SSL_read case for now for initial testing.

-- 
John Baldwin