Getting started with ktls

[Alan Somers]

On Thu, Mar 11, 2021 at 11:49 AM John Baldwin <jhb at freebsd.org> wrote:

> On 3/10/21 4:18 PM, Alan Somers wrote:
> > I'm trying to make ktls work with "zfs send/recv" to substantially reduce
> > the CPU utilization of applications like zrepl.  But I have a few
> questions:
> >
> > * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by a
> > successful set of the TCP_TXTLS_ENABLE socket option", but the "Supported
> > Libraries" section says "Applications using a supported library should
> > generally work with ktls without any changes".  These sentences seem to
> be
> > contradictory.  I think it means that the TCP_TXTLS_ENABLE option is
> > necessary, but OpenSSL sets it automatically?
>
> Yes, you can do it by hand if you want but you'd have to do all the key
> exchange by hand as well.
>
> > * When using OpenSSL, the library will automatically call setsockopt(_,
> > TCP_TXTLS_ENABLE).  But it swallows the error, if any.  How is an
> > application to tell if ktls is enabled on a particular socket or OpenSSL
> > session?
>
> BIO_get_ktls_send() and BIO_get_ktls_recv() on the write and read BIO's of
> the connection, respectively.
>
> > * From experiment, I can see that OpenSSL attempts to set
> > TCP_TXTLS_ENABLE.  But it doesn't try to set TCP_RXTLS_ENABLE.  Why not?
> >  From reading ktls_start and ossl_statem_server_post_work, it looks like
> > maybe a single socket cannot have ktls enabled for both sending and
> > receiving at the same time.  Is that true?
>
> Neither FreeBSD nor OpenSSL yet support RX offload on TLS 1.3.  If you use
> TLS 1.2 you will get KTLS in both directions (or if you use TLS 1.1 with
> TOE offload on a Chelsio T6).
>
> --
> John Baldwin
>

Switching to TLS 1.2 turned out to be key.  Once I did that, ... it just
worked.  I was expecting to need minor changes throughout the kernel and
libzfs.  However, that wasn't necessary.  Here is my proof-of-concept
program.  So far only the recv path is implemented.  I'll probably
implement the send path too, but I'm not currently planning to integrate
this into any open-source application. Thanks for all the help!

https://github.com/asomers/freebsd-src/tree/ktls-zfs/tools/zfs-ktls

-Alan