[Bug 295942] SSL_OP_ENABLE_KTLS_TX_ZEROCOPY_SENDFILE data corruption on files > 128KB

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=295942

Andrew Gallatin <gallatin@FreeBSD.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |gallatin@FreeBSD.org

--- Comment #1 from Andrew Gallatin <gallatin@FreeBSD.org> ---
I tried to reproduce this.  I installed 15R into a bhyve VM and selected e1000
as the nic (which is as close as I can get to igb, and they both use iflib, so
I don't think that matters).

I was unable to reproduce it.  My conf file looks like:
root@15R:/home/gallatin # egrep -v ^# /usr/local/etc/lighttpd/lighttpd.conf
include "/usr/local/etc/lighttpd/lighttpd*annotated.conf"
include "/usr/local/etc/lighttpd/conf-enabled/*.conf"


server.modules += ( "mod_openssl" )

$SERVER["socket"] == ":443" {
    ssl.engine = "enable"
    ssl.privkey = "/usr/local/etc/lighttpd/ssl/server.key"
    ssl.pemfile = "/usr/local/etc/lighttpd/ssl/server.crt"
}

And after transferring a large file (/boot/kernel/kernel copied to the document
root) many times, I don't see any issues.

One thing I noticed is that I see separate_output elevated in my counters,
which indicates the server is using sendfile (confirmed via truss).  The
reporters counters show inplace elevated:

root@15R:/home/gallatin # sysctl kern.ipc.tls | grep -v ': 0'
kern.ipc.tls.ifnet.permitted: 1
kern.ipc.tls.sw.gcm: 84
kern.ipc.tls.max_reclaim: 1024
kern.ipc.tls.sw_buffer_cache: 1
kern.ipc.tls.cbc_enable: 1
kern.ipc.tls.enable: 1
kern.ipc.tls.ifnet_max_rexmit_pct: 2
kern.ipc.tls.maxlen: 16384
kern.ipc.tls.stats.ocf.separate_output: 139557
kern.ipc.tls.stats.ocf.inplace: 312
kern.ipc.tls.stats.ocf.tls13_gcm_encrypts: 139869
kern.ipc.tls.stats.ocf.tls13_gcm_decrypts: 935
kern.ipc.tls.stats.active: 84
kern.ipc.tls.stats.enable_calls: 156
kern.ipc.tls.stats.offload_total: 156
kern.ipc.tls.stats.sw_tx_inqueue: 1
kern.ipc.tls.stats.threads: 8


Dirk:  Are you serving from a ZFS or UFS fs?  If you truss the web server while
serving (and ktls enabled), do you see sendfile?  Eg:

root@15R:/home/gallatin # truss -cp `pgrep httpd`
^Csyscall                     seconds   calls  errors
close                   0.000044609       2       0
shutdown                0.003182343       2       0
sendfile                0.011392569     240      26
pread                   0.000103809       6       0
fstatat                 0.000041602       2       0
recvmsg                 0.000379371      23       1
sendmsg                 0.000223300       4       0
setsockopt              0.000085522       4       0
write                   0.000330193      12       2
getpid                  0.000484688      18       0
kevent                  1.888839601     257       0
read                    0.000713765      22       6
accept4                 0.000378697       4       2
                      ------------- ------- -------
                        1.906200069     596      37

-- 
You are receiving this mail because:
You are the assignee for the bug.