From nobody Mon Nov 10 15:51:29 2025 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4d4vKZ05kPz6GtCl; Mon, 10 Nov 2025 15:51:30 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4d4vKY4z9Rz449h; Mon, 10 Nov 2025 15:51:29 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1762789889; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=WJ6gfo+DTSMsKf9P2LrOGX2ehXQAXb84OR72PR6ptWY=; b=gleSQh48UKZB5gyfZmFaiIRjiqFuQ/uUWe5876sc4C5Ubk/HHUCwEEI0d50P16cnxpJW3m qyKNDUJg1FbulFrgFp07ct17kflAeLYC+FdhuJi6r4BWWytj0y+JzDNfO6kTCJwSpYkp/T WrWrd//ok1prpoyEezqvDB5zzRihWNG4SiPLZq/s7uN3cVDdTsxR8NaG6nCPAKqGFZYNPm gxPUTSWw92BTtmEK0IOYcBu3qrRJH7EduYG0vuhuC9Owu65BapsbHzf9RuzE2q+87iSk6a fTn6T2ZWNjxV4lwXCdIz7uX+RS70M3y5kxR/h3jVEfVyOMpfNfNisDlrRccPTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1762789889; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=WJ6gfo+DTSMsKf9P2LrOGX2ehXQAXb84OR72PR6ptWY=; b=jqk0/qzTKfdd7R1IatUrHniY0od5dvwjn2MnP4AEqJSMJMWFRKQppiu+armz+MLgS3zWRJ Xx+JdVmTex+/ykdkyqPaXzMzZxuW7ZsI+txJ+nK09g2RW6jp9pPrLJgASqSic9HwoB0kHB 3M2iGAx42slBOi/ggIF/BdWJmAK/wgPLqDPKik3GKCsEJlJSsFCLHxJXrkIef9PIdIFXLR RebZDPwie3ksxMbslWLeODknJJ0/bEcLHAT4yEKNOF0SuORAnTs/fgvZkFftd5Nt5dqAae 7n3PDmzddiX3JoCrkbynLF2ZDY2NyyxIfrbllPuyRkNIL16IhDtQe5QvWDnH6w== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1762789889; a=rsa-sha256; cv=none; b=lyBfxFPl0uAjWn/gt5IifRv6XZtxDs3I4wtNeSdm+pHAeQV1sYQjh8qqybkqs60XXFrIQR Npc1k1NtzgV6jlej1cOBfTGzNJOZs7gJYi77mYH8kyWqjszqX0jgtfjK2Nhi/tvx7PDVdx jyL+5qiozDoQ7xL4dgNbQg05uh6kAqH5LW+eN7CUxEsUOD2mPKHXxGNgv085RVjfP2S0aw 6S4EYHVBMkI7jop2/LIxuNnRPh2hHoYlbuAl9hgpNzPWDD4zeSUbvFlxQwYy3FTNSG1C/O b8Fg9k22g7Awa/yjELVtAgVfYEdBV1ozXk7qcIMPxmETPAMFqmPqzDlmmJWhqQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4d4vKY4RKhz7qr; Mon, 10 Nov 2025 15:51:29 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 5AAFpT34048223; Mon, 10 Nov 2025 15:51:29 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 5AAFpTcb048220; Mon, 10 Nov 2025 15:51:29 GMT (envelope-from git) Date: Mon, 10 Nov 2025 15:51:29 GMT Message-Id: <202511101551.5AAFpTcb048220@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: John Baldwin Subject: git: 71d82199a111 - main - cxgbe: Add support to the base driver for NVMe/TCP PDU offload List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 71d82199a111af67cba73e32a27a900d74c1a1cc Auto-Submitted: auto-generated The branch main has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=71d82199a111af67cba73e32a27a900d74c1a1cc commit 71d82199a111af67cba73e32a27a900d74c1a1cc Author: John Baldwin AuthorDate: 2025-11-10 15:50:48 +0000 Commit: John Baldwin CommitDate: 2025-11-10 15:50:48 +0000 cxgbe: Add support to the base driver for NVMe/TCP PDU offload - Adds various per-queue counters similar to iSCSI PDU offload as well as a hook in the adapter softc for a reference to the NVMe/TCP softc. - Instruct the firmware to include a DDP indicator in the status field for NVMe/TCP PDU completion messages. This flag indicates if the payload data for a PDU has been received in the free list or if it was placed directly into a kernel I/O data buffer via DDP. Sponsored by: Chelsio Communications --- sys/dev/cxgbe/adapter.h | 11 ++++++++ sys/dev/cxgbe/t4_main.c | 16 ++++++++++++ sys/dev/cxgbe/t4_sge.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 93 insertions(+), 2 deletions(-) diff --git a/sys/dev/cxgbe/adapter.h b/sys/dev/cxgbe/adapter.h index 57618b1ceb15..0946c3110817 100644 --- a/sys/dev/cxgbe/adapter.h +++ b/sys/dev/cxgbe/adapter.h @@ -723,6 +723,16 @@ struct sge_ofld_rxq { uint64_t rx_iscsi_padding_errors; uint64_t rx_iscsi_header_digest_errors; uint64_t rx_iscsi_data_digest_errors; + counter_u64_t rx_nvme_ddp_setup_ok; + counter_u64_t rx_nvme_ddp_setup_no_stag; + counter_u64_t rx_nvme_ddp_setup_error; + counter_u64_t rx_nvme_ddp_pdus; + counter_u64_t rx_nvme_ddp_octets; + counter_u64_t rx_nvme_fl_pdus; + counter_u64_t rx_nvme_fl_octets; + counter_u64_t rx_nvme_invalid_headers; + counter_u64_t rx_nvme_header_digest_errors; + counter_u64_t rx_nvme_data_digest_errors; uint64_t rx_aio_ddp_jobs; uint64_t rx_aio_ddp_octets; u_long rx_toe_tls_records; @@ -1000,6 +1010,7 @@ struct adapter { void *iwarp_softc; /* (struct c4iw_dev *) */ struct iw_tunables iwt; void *iscsi_ulp_softc; /* (struct cxgbei_data *) */ + void *nvme_ulp_softc; /* (struct nvmf_che_adapter *) */ struct l2t_data *l2t; /* L2 table */ struct smt_data *smt; /* Source MAC Table */ struct tid_info tids; diff --git a/sys/dev/cxgbe/t4_main.c b/sys/dev/cxgbe/t4_main.c index 2f6eb6062c20..6133d810c003 100644 --- a/sys/dev/cxgbe/t4_main.c +++ b/sys/dev/cxgbe/t4_main.c @@ -13009,6 +13009,22 @@ clear_stats(struct adapter *sc, u_int port_id) ofld_rxq->rx_iscsi_ddp_octets = 0; ofld_rxq->rx_iscsi_fl_pdus = 0; ofld_rxq->rx_iscsi_fl_octets = 0; + counter_u64_zero( + ofld_rxq->rx_nvme_ddp_setup_ok); + counter_u64_zero( + ofld_rxq->rx_nvme_ddp_setup_no_stag); + counter_u64_zero( + ofld_rxq->rx_nvme_ddp_setup_error); + counter_u64_zero(ofld_rxq->rx_nvme_ddp_pdus); + counter_u64_zero(ofld_rxq->rx_nvme_ddp_octets); + counter_u64_zero(ofld_rxq->rx_nvme_fl_pdus); + counter_u64_zero(ofld_rxq->rx_nvme_fl_octets); + counter_u64_zero( + ofld_rxq->rx_nvme_invalid_headers); + counter_u64_zero( + ofld_rxq->rx_nvme_header_digest_errors); + counter_u64_zero( + ofld_rxq->rx_nvme_data_digest_errors); ofld_rxq->rx_aio_ddp_jobs = 0; ofld_rxq->rx_aio_ddp_octets = 0; ofld_rxq->rx_toe_tls_records = 0; diff --git a/sys/dev/cxgbe/t4_sge.c b/sys/dev/cxgbe/t4_sge.c index 8cd5b0543d8a..4932a3abdeca 100644 --- a/sys/dev/cxgbe/t4_sge.c +++ b/sys/dev/cxgbe/t4_sge.c @@ -852,6 +852,11 @@ t4_tweak_chip_settings(struct adapter *sc) /* We use multiple DDP page sizes both in plain-TOE and ISCSI modes. */ m = v = F_TDDPTAGTCB | F_ISCSITAGTCB; + if (sc->nvmecaps != 0) { + /* Request DDP status bit for NVMe PDU completions. */ + m |= F_NVME_TCP_DDP_VAL_EN; + v |= F_NVME_TCP_DDP_VAL_EN; + } t4_set_reg_field(sc, A_ULP_RX_CTL, m, v); m = V_INDICATESIZE(M_INDICATESIZE) | F_REARMDDPOFFSET | @@ -4170,6 +4175,20 @@ alloc_ofld_rxq(struct vi_info *vi, struct sge_ofld_rxq *ofld_rxq, int idx, ofld_rxq->rx_iscsi_ddp_setup_ok = counter_u64_alloc(M_WAITOK); ofld_rxq->rx_iscsi_ddp_setup_error = counter_u64_alloc(M_WAITOK); + ofld_rxq->rx_nvme_ddp_setup_ok = counter_u64_alloc(M_WAITOK); + ofld_rxq->rx_nvme_ddp_setup_no_stag = + counter_u64_alloc(M_WAITOK); + ofld_rxq->rx_nvme_ddp_setup_error = + counter_u64_alloc(M_WAITOK); + ofld_rxq->rx_nvme_ddp_octets = counter_u64_alloc(M_WAITOK); + ofld_rxq->rx_nvme_ddp_pdus = counter_u64_alloc(M_WAITOK); + ofld_rxq->rx_nvme_fl_octets = counter_u64_alloc(M_WAITOK); + ofld_rxq->rx_nvme_fl_pdus = counter_u64_alloc(M_WAITOK); + ofld_rxq->rx_nvme_invalid_headers = counter_u64_alloc(M_WAITOK); + ofld_rxq->rx_nvme_header_digest_errors = + counter_u64_alloc(M_WAITOK); + ofld_rxq->rx_nvme_data_digest_errors = + counter_u64_alloc(M_WAITOK); ofld_rxq->ddp_buffer_alloc = counter_u64_alloc(M_WAITOK); ofld_rxq->ddp_buffer_reuse = counter_u64_alloc(M_WAITOK); ofld_rxq->ddp_buffer_free = counter_u64_alloc(M_WAITOK); @@ -4207,6 +4226,16 @@ free_ofld_rxq(struct vi_info *vi, struct sge_ofld_rxq *ofld_rxq) MPASS(!(ofld_rxq->iq.flags & IQ_SW_ALLOCATED)); counter_u64_free(ofld_rxq->rx_iscsi_ddp_setup_ok); counter_u64_free(ofld_rxq->rx_iscsi_ddp_setup_error); + counter_u64_free(ofld_rxq->rx_nvme_ddp_setup_ok); + counter_u64_free(ofld_rxq->rx_nvme_ddp_setup_no_stag); + counter_u64_free(ofld_rxq->rx_nvme_ddp_setup_error); + counter_u64_free(ofld_rxq->rx_nvme_ddp_octets); + counter_u64_free(ofld_rxq->rx_nvme_ddp_pdus); + counter_u64_free(ofld_rxq->rx_nvme_fl_octets); + counter_u64_free(ofld_rxq->rx_nvme_fl_pdus); + counter_u64_free(ofld_rxq->rx_nvme_invalid_headers); + counter_u64_free(ofld_rxq->rx_nvme_header_digest_errors); + counter_u64_free(ofld_rxq->rx_nvme_data_digest_errors); counter_u64_free(ofld_rxq->ddp_buffer_alloc); counter_u64_free(ofld_rxq->ddp_buffer_reuse); counter_u64_free(ofld_rxq->ddp_buffer_free); @@ -4218,12 +4247,12 @@ static void add_ofld_rxq_sysctls(struct sysctl_ctx_list *ctx, struct sysctl_oid *oid, struct sge_ofld_rxq *ofld_rxq) { - struct sysctl_oid_list *children; + struct sysctl_oid_list *children, *top; if (ctx == NULL || oid == NULL) return; - children = SYSCTL_CHILDREN(oid); + top = children = SYSCTL_CHILDREN(oid); SYSCTL_ADD_U64(ctx, children, OID_AUTO, "rx_aio_ddp_jobs", CTLFLAG_RD, &ofld_rxq->rx_aio_ddp_jobs, 0, "# of aio_read(2) jobs completed via DDP"); @@ -4280,6 +4309,41 @@ add_ofld_rxq_sysctls(struct sysctl_ctx_list *ctx, struct sysctl_oid *oid, SYSCTL_ADD_U64(ctx, children, OID_AUTO, "data_digest_errors", CTLFLAG_RD, &ofld_rxq->rx_iscsi_data_digest_errors, 0, "# of PDUs with invalid data digests"); + + oid = SYSCTL_ADD_NODE(ctx, top, OID_AUTO, "nvme", + CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "TOE NVMe statistics"); + children = SYSCTL_CHILDREN(oid); + + SYSCTL_ADD_COUNTER_U64(ctx, children, OID_AUTO, "ddp_setup_ok", + CTLFLAG_RD, &ofld_rxq->rx_nvme_ddp_setup_ok, + "# of times DDP buffer was setup successfully"); + SYSCTL_ADD_COUNTER_U64(ctx, children, OID_AUTO, "ddp_setup_no_stag", + CTLFLAG_RD, &ofld_rxq->rx_nvme_ddp_setup_no_stag, + "# of times STAG was not available for DDP buffer setup"); + SYSCTL_ADD_COUNTER_U64(ctx, children, OID_AUTO, "ddp_setup_error", + CTLFLAG_RD, &ofld_rxq->rx_nvme_ddp_setup_error, + "# of times DDP buffer setup failed"); + SYSCTL_ADD_COUNTER_U64(ctx, children, OID_AUTO, "ddp_octets", + CTLFLAG_RD, &ofld_rxq->rx_nvme_ddp_octets, + "# of octets placed directly"); + SYSCTL_ADD_COUNTER_U64(ctx, children, OID_AUTO, "ddp_pdus", + CTLFLAG_RD, &ofld_rxq->rx_nvme_ddp_pdus, + "# of PDUs with data placed directly"); + SYSCTL_ADD_COUNTER_U64(ctx, children, OID_AUTO, "fl_octets", + CTLFLAG_RD, &ofld_rxq->rx_nvme_fl_octets, + "# of data octets delivered in freelist"); + SYSCTL_ADD_COUNTER_U64(ctx, children, OID_AUTO, "fl_pdus", + CTLFLAG_RD, &ofld_rxq->rx_nvme_fl_pdus, + "# of PDUs with data delivered in freelist"); + SYSCTL_ADD_COUNTER_U64(ctx, children, OID_AUTO, "invalid_headers", + CTLFLAG_RD, &ofld_rxq->rx_nvme_invalid_headers, + "# of PDUs with invalid header field"); + SYSCTL_ADD_COUNTER_U64(ctx, children, OID_AUTO, "header_digest_errors", + CTLFLAG_RD, &ofld_rxq->rx_nvme_header_digest_errors, + "# of PDUs with invalid header digests"); + SYSCTL_ADD_COUNTER_U64(ctx, children, OID_AUTO, "data_digest_errors", + CTLFLAG_RD, &ofld_rxq->rx_nvme_data_digest_errors, + "# of PDUs with invalid data digests"); } #endif