KTLS with zfs recv

[Rick Macklem]

Warner Losh wrote:
>On Fri, Feb 26, 2021 at 11:16 AM Rodney W. Grimes <
>freebsd-rwg at gndrsh.dnsmgr.net> wrote:
>
>> > On Fri, Feb 26, 2021 at 9:24 AM Rodney W. Grimes <
>> > freebsd-rwg at gndrsh.dnsmgr.net> wrote:
>> >
>> > > > My understanding is that KTLS works very well with OpenSSL for
>> sending,
>> > > but
>> > > > not as well for receiving, because there's nothing like a recvfile
>> > > > syscall.  However, it works great for both send and receive with NFS,
>> > > where
>> > > > all the data remains in the kernel. What about zfs recv?  A very
>> common
>> > > > pattern is for an application to read from an SSL socket and then
>> pipe
>> > > the
>> > > > data to zfs recv. For example, zrepl does that.  Could zfs recv
>> instead
>> > > > read directly from the KTLS socket, bypassing userspace?  That could
>> > > > potentially save a _lot_ of cycles for a _lot_ of people.
>> > >
>> > > I did some patches and a short presentation at BSDCan that basically
>> > > shoves the whole zfs send and zfs recv process into the kernel, ie
>> > > it opens the sockets up, makes the connections, then the socket
>> > > is passed into the kernel(s) and it all runs in kernel mode.
>> > >
>> > >
>> > >
>> https://www.bsdcan.org/2018/schedule/attachments/479_BSDCan-2018-zfs-send.pdf
>> > >
>> > > A few things need fixed like reversing who does the listen for
>> > > security reasons, but this feature is probably ready for prime
>> > > time.
>> > >
>> > > > -Alan
>> > >
>> > > --
>> > > Rod Grimes
>> > > rgrimes at freebsd.org
>> >
>> >
>> > That looks potentially useful, but it doesn't use encryption.  Would it
>> > work if the socket had been opened by openssl with ktls?
>>
>> Yes, it should.  Internally the zfs send and recv code just does reads
>> and writes to the socket, so what ever you setup for "connected" sockets
>> should work.
>>
>
>Yea, KTLS generally wants userland to do the initial negotiation and share
>the connection state before doing the bulk encryption in the kernel...
Yes. KTLS only handles application data records. The TLS handshake must
be done in userspace.

Handling of non-application data records received after the initial handshake
also needs to be handled in userspace. All the userspace program does is an
SSL_read() with a len argument == 0. This will always return an error, but only
after it has read the non-application data records off the head of the socket's
receive queue. The kernel code needs to tell the userspace program to do this
somehow. The nfs-over-tls code does this via an upcall RPC to the userspace
daemon (rpc.tlsclntd or rpc.tlsservd).

I added MSG_TLSAPPDATA which can be used as an argument to soreceive()
to tell it to return an error when there is a non-application data record at the
head of the socket's receive queue, to facilitate this.

You do normal sosend()/soreceive() on the socket with unencrypted data,
although each soreceive() will give you a ctrlmsg mbuf with TLS record
info, if you care (when you use MSG_TLSAPPDATA, they're all just application
data records). Each soreceive() gives you one application data record, but
the code probably won't care.

Finally, when done transferring data, userspace probably wants to do
SSL_shutdown() to do the peer reset (although if/when you need to do
peer resets seems to be a bit sketchy in the TLS game).

rpc.tlsclntd.c and rpc.tlscommon.c probably gives you some useful hints.

It sounds worthwhile to explore this.

Good luck with it, rick

Warner
_______________________________________________
freebsd-current at freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"