From nobody Tue Nov 23 23:12:55 2021 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 495A818A1B48; Tue, 23 Nov 2021 23:13:00 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HzKgD6sPNz3vmX; Tue, 23 Nov 2021 23:12:56 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 807A11F149; Tue, 23 Nov 2021 23:12:55 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 1ANNCtse037711; Tue, 23 Nov 2021 23:12:55 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 1ANNCtpu037710; Tue, 23 Nov 2021 23:12:55 GMT (envelope-from git) Date: Tue, 23 Nov 2021 23:12:55 GMT Message-Id: <202111232312.1ANNCtpu037710@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: John Baldwin Subject: git: ba6b771d1732 - stable/13 - ktls: Ensure FIFO encryption order for TLS 1.0. List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: ba6b771d1732eda0546d187b1397b1bcded3208d Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1637709177; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=e9ilAIy8JKe+rjTySFbBFfagIh2m2moi/eQerqLHp6w=; b=eAOpfi4rl3ibPXADhdadePnbjYgr8zGLhJkrYfoeYwD/N9prvbzZ0wbFmARoZJPPmK3i2C 2cspeI6oGirL5lS/c92gpCeCAbpBz9gg0NoVQBJZXiC/nTqJ0757ZvMEXHeCuD5olGsJw9 tiAVs05JOHhY/KGgx+tFja/4WJV1BAlWqh/QWQs7wxUMy15N82/153VE2ugKGGfxIazAIo ml6uz3xTBSeQx7XwD/qIENSuT2v7AapEHR1W4bR8tktw5uvGb7Z7c4CnBB+WnA6B5Vej67 RXRoI2HkVG0nwgKYpQsZ4AjXU0d4wSb/+y2WfeFHk1uLVn8gM3NQs0Qwb8QUrA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1637709177; a=rsa-sha256; cv=none; b=mJHoLB4POJZvkHjc6JD4Oll/uTs/YbPtviMVl6gZLubY3xXs5+eWeEEUIAUBndzAsW0n01 V0b4NY823gpjFtfz1+S5Wtjhmhd7KaSHdy55rUQnog7GzVyzEkUSK1EmG6Xd6tqPtAsQLb nNkA9xEV18f93sk5uJRgI+kiYmHyA4piMmAkqlwCOXxDH6OfWr74uiVkWcr/RhfpaQJjwB LBKhyFvMAYdLq7sFDkJMT6Rh0cgvTJWaiWZkGEWCTApdFX2sDWYirMT6OkDWGBgfG8HLbi etMH4Y2o4IS8LPMvsFri65u1/BrNXrJrIlgwlGQc+R+lUDSgUsOYDHNqXaK4Dw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=ba6b771d1732eda0546d187b1397b1bcded3208d commit ba6b771d1732eda0546d187b1397b1bcded3208d Author: John Baldwin AuthorDate: 2021-10-13 19:30:15 +0000 Commit: John Baldwin CommitDate: 2021-11-23 23:11:44 +0000 ktls: Ensure FIFO encryption order for TLS 1.0. TLS 1.0 records are encrypted as one continuous CBC chain where the last block of the previous record is used as the IV for the next record. As a result, TLS 1.0 records cannot be encrypted out of order but must be encrypted as a FIFO. If the later pages of a sendfile(2) request complete before the first pages, then TLS records can be encrypted out of order. For TLS 1.1 and later this is fine, but this can break for TLS 1.0. To cope, add a queue in each TLS session to hold TLS records that contain valid unencrypted data but are waiting for an earlier TLS record to be encrypted first. - In ktls_enqueue(), check if a TLS record being queued is the next record expected for a TLS 1.0 session. If not, it is placed in sorted order in the pending_records queue in the TLS session. If it is the next expected record, queue it for SW encryption like normal. In addition, check if this new record (really a potential batch of records) was holding up any previously queued records in the pending_records queue. Any of those records that are now in order are also placed on the queue for SW encryption. - In ktls_destroy(), free any TLS records on the pending_records queue. These mbufs are marked M_NOTREADY so were not freed when the socket buffer was purged in sbdestroy(). Instead, they must be freed explicitly. Reviewed by: gallatin, markj Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D32381 (cherry picked from commit 9f03d2c00167c8047416e0048e3b7f89d73baf8e) --- sys/kern/uipc_ktls.c | 113 +++++++++++++++++++++++++++++++++++++++++++++++++-- sys/sys/ktls.h | 5 +++ 2 files changed, 115 insertions(+), 3 deletions(-) diff --git a/sys/kern/uipc_ktls.c b/sys/kern/uipc_ktls.c index 73915600779c..e9585d06841c 100644 --- a/sys/kern/uipc_ktls.c +++ b/sys/kern/uipc_ktls.c @@ -138,6 +138,11 @@ static COUNTER_U64_DEFINE_EARLY(ktls_tasks_active); SYSCTL_COUNTER_U64(_kern_ipc_tls, OID_AUTO, tasks_active, CTLFLAG_RD, &ktls_tasks_active, "Number of active tasks"); +static COUNTER_U64_DEFINE_EARLY(ktls_cnt_tx_pending); +SYSCTL_COUNTER_U64(_kern_ipc_tls_stats, OID_AUTO, sw_tx_pending, CTLFLAG_RD, + &ktls_cnt_tx_pending, + "Number of TLS 1.0 records waiting for earlier TLS records"); + static COUNTER_U64_DEFINE_EARLY(ktls_cnt_tx_queued); SYSCTL_COUNTER_U64(_kern_ipc_tls_stats, OID_AUTO, sw_tx_inqueue, CTLFLAG_RD, &ktls_cnt_tx_queued, @@ -574,6 +579,9 @@ ktls_create_session(struct socket *so, struct tls_enable *en, case CRYPTO_SHA1_HMAC: if (en->tls_vminor == TLS_MINOR_VER_ZERO) { /* Implicit IV, no nonce. */ + tls->sequential_records = true; + tls->next_seqno = be64dec(en->rec_seq); + STAILQ_INIT(&tls->pending_records); } else { tls->params.tls_hlen += AES_BLOCK_LEN; } @@ -1481,6 +1489,20 @@ ktls_destroy(struct ktls_session *tls) { struct rm_priotracker prio; + if (tls->sequential_records) { + struct mbuf *m, *n; + int page_count; + + STAILQ_FOREACH_SAFE(m, &tls->pending_records, m_epg_stailq, n) { + page_count = m->m_epg_enc_cnt; + while (page_count > 0) { + KASSERT(page_count >= m->m_epg_nrdy, + ("%s: too few pages", __func__)); + page_count -= m->m_epg_nrdy; + m = m_free(m); + } + } + } ktls_cleanup(tls); if (tls->be != NULL && ktls_allow_unload) { rm_rlock(&ktls_backends_lock, &prio); @@ -1982,10 +2004,29 @@ ktls_enqueue_to_free(struct mbuf *m) wakeup(wq); } +/* Number of TLS records in a batch passed to ktls_enqueue(). */ +static u_int +ktls_batched_records(struct mbuf *m) +{ + int page_count, records; + + records = 0; + page_count = m->m_epg_enc_cnt; + while (page_count > 0) { + records++; + page_count -= m->m_epg_nrdy; + m = m->m_next; + } + KASSERT(page_count == 0, ("%s: mismatched page count", __func__)); + return (records); +} + void ktls_enqueue(struct mbuf *m, struct socket *so, int page_count) { + struct ktls_session *tls; struct ktls_wq *wq; + int queued; bool running; KASSERT(((m->m_flags & (M_EXTPG | M_NOTREADY)) == @@ -2003,14 +2044,80 @@ ktls_enqueue(struct mbuf *m, struct socket *so, int page_count) */ m->m_epg_so = so; - wq = &ktls_wq[m->m_epg_tls->wq_index]; + queued = 1; + tls = m->m_epg_tls; + wq = &ktls_wq[tls->wq_index]; mtx_lock(&wq->mtx); - STAILQ_INSERT_TAIL(&wq->m_head, m, m_epg_stailq); + if (__predict_false(tls->sequential_records)) { + /* + * For TLS 1.0, records must be encrypted + * sequentially. For a given connection, all records + * queued to the associated work queue are processed + * sequentially. However, sendfile(2) might complete + * I/O requests spanning multiple TLS records out of + * order. Here we ensure TLS records are enqueued to + * the work queue in FIFO order. + * + * tls->next_seqno holds the sequence number of the + * next TLS record that should be enqueued to the work + * queue. If this next record is not tls->next_seqno, + * it must be a future record, so insert it, sorted by + * TLS sequence number, into tls->pending_records and + * return. + * + * If this TLS record matches tls->next_seqno, place + * it in the work queue and then check + * tls->pending_records to see if any + * previously-queued records are now ready for + * encryption. + */ + if (m->m_epg_seqno != tls->next_seqno) { + struct mbuf *n, *p; + + p = NULL; + STAILQ_FOREACH(n, &tls->pending_records, m_epg_stailq) { + if (n->m_epg_seqno > m->m_epg_seqno) + break; + p = n; + } + if (n == NULL) + STAILQ_INSERT_TAIL(&tls->pending_records, m, + m_epg_stailq); + else if (p == NULL) + STAILQ_INSERT_HEAD(&tls->pending_records, m, + m_epg_stailq); + else + STAILQ_INSERT_AFTER(&tls->pending_records, p, m, + m_epg_stailq); + mtx_unlock(&wq->mtx); + counter_u64_add(ktls_cnt_tx_pending, 1); + return; + } + + tls->next_seqno += ktls_batched_records(m); + STAILQ_INSERT_TAIL(&wq->m_head, m, m_epg_stailq); + + while (!STAILQ_EMPTY(&tls->pending_records)) { + struct mbuf *n; + + n = STAILQ_FIRST(&tls->pending_records); + if (n->m_epg_seqno != tls->next_seqno) + break; + + queued++; + STAILQ_REMOVE_HEAD(&tls->pending_records, m_epg_stailq); + tls->next_seqno += ktls_batched_records(n); + STAILQ_INSERT_TAIL(&wq->m_head, n, m_epg_stailq); + } + counter_u64_add(ktls_cnt_tx_pending, -(queued - 1)); + } else + STAILQ_INSERT_TAIL(&wq->m_head, m, m_epg_stailq); + running = wq->running; mtx_unlock(&wq->mtx); if (!running) wakeup(wq); - counter_u64_add(ktls_cnt_tx_queued, 1); + counter_u64_add(ktls_cnt_tx_queued, queued); } static __noinline void diff --git a/sys/sys/ktls.h b/sys/sys/ktls.h index 3cde75f9edf6..e9cae34035c6 100644 --- a/sys/sys/ktls.h +++ b/sys/sys/ktls.h @@ -208,6 +208,11 @@ struct ktls_session { struct task reset_tag_task; struct inpcb *inp; bool reset_pending; + bool sequential_records; + + /* Only used for TLS 1.0. */ + uint64_t next_seqno; + STAILQ_HEAD(, mbuf) pending_records; } __aligned(CACHE_LINE_SIZE); void ktls_check_rx(struct sockbuf *sb);