From nobody Wed Sep 16 14:13:14 2026 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hlLT66KQKz6rWqy for ; Wed, 16 Sep 2026 14:13:14 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hlLT643nFz3MqJ for ; Wed, 16 Sep 2026 14:13:14 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1789567994; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=jVlpVNB1RgTG2+pP+dCW0uJhZmYgXVyBuXEfSQHjI6E=; b=tGBLrMTQp1YCUwnsIAFaAqACPOydTB56XhKHJoCv/qrnQ1hSpdUHh9ICVuj16pjNofc7f7 kfO+5tLeSi7YWNIT94ueO1Qv2GwmtdEh5c24ESPNqtMcDjD4SA5Hm1f33rRihxCoUZtMNs nP8qprHHwLcEblQT/+tSrboQv78KceVLdyxik2pxYUV4UQVy1X3QsdISA2KsihfJTgR4Zo RCrSXOocFxH7CDCOeotdClB7aYkgZ+BC5WdHW9gfLziZAbBEW3udzRDfGC30aVhMB2MA9n FHm1hNFfsq6rjf7IIMcoEnIlHE5gFb9s+4ToEq25b+q6lRXbhq7KwOBfDdKIWA== ARC-Seal: i=1; a=rsa-sha256; d=freebsd.org; s=dkim; cv=none; t=1789567994; b=ufDLeqH/0pP1Ajnu06+yuLfzs5PNuMW062GeR9KAVSim4jTvcLkHG+jEFP2wBExd7KxrBH KwiWqsraOwIV86MBZMbewRj2Yit+y9mTVpCK4hQhulQuJF1gd4c8A/Fu1jJdmx89GPeLmh zzTSBWN2upVPXehMayFyBZPvfX1HROiYliUfWhpq2RPUT9hey9oDrQF0OdsxROBNujfepd 0/7n5tJuDwi8bj4YV2X1s/Tpy49Cp6G9F420OCLNow1bSfQ+VW5qBWkjVmRHKTAqlQk+KB MKq66g7ttU0cioYZOH8AZ7+LDvnm6BPg/WQGZRofVjFb399rJsY3Um768UVXkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1789567994; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=jVlpVNB1RgTG2+pP+dCW0uJhZmYgXVyBuXEfSQHjI6E=; b=ai/PKeWTcFG7V8C/bSfzgILNvpeRBHvAD5JwLHI4cYHrtUUlAGFMYS//1P3FWFobtDhriu 44Xf4TRsS76k+977ehDUctFD/Nm/AinfI/PPYLwIo8GaL6YoZ/Fqb8LqldrqArtvrs2EQg 3N9XDMqk9Q+Z0MuIoxZ25V3OUqWTR3h8zF06+QRYWOcLcIchPWwzKKGLNJLgq6Twtahbl5 1bYDN2vDH8qtxte+beuCtlI0lXDkM8h+GwUEKmwzRBzJgN5n/TOmvXrOMrte1O6D2/GoZ4 5OmieHSTmyFadkgrgVObYQv6/ZsBERzjzRiJO5i7FvWEstErhCRVqYuZXRvbnQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hlLT632V0zqWr for ; Wed, 16 Sep 2026 14:13:14 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 35fd2 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Wed, 16 Sep 2026 14:13:14 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Cc: Vag Singh From: Konstantin Belousov Subject: git: a5a8f98eb7a4 - main - IB/mlx5: Add tunneling offloads support List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kib X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: a5a8f98eb7a4494e03b0052e60d1010fd433af44 Auto-Submitted: auto-generated Date: Wed, 16 Sep 2026 14:13:14 +0000 Message-Id: <6aaaa3fa.35fd2.26426fc1@gitrepo.freebsd.org> The branch main has been updated by kib: URL: https://cgit.FreeBSD.org/src/commit/?id=a5a8f98eb7a4494e03b0052e60d1010fd433af44 commit a5a8f98eb7a4494e03b0052e60d1010fd433af44 Author: Vag Singh AuthorDate: 2025-08-14 06:47:04 +0000 Commit: Konstantin Belousov CommitDate: 2026-09-16 14:12:09 +0000 IB/mlx5: Add tunneling offloads support [PATCH 22/31] FreeBSD OFED support for DPDK MLX5 PMD a) The device can support receive Stateless Offloads for the inner packet's fields only when the packet is processed by TIR which is enabled to support tunneling. Otherwise, the device treats the packet as an ordinary non-tunneling packet and receive offloads can be done only for the outer packet's field. In order to enable receive Stateless Offloading support for incoming tunneling traffic the TIR should be created with tunneled_offload_en. Tunneling offloads is supported only be raw ethernet QP. This patch includes: - New QP creation flag for tunneling offloads. - Reports device capabilities. b) IB/mlx5: Add support for RSS on the inner packet Some user space application would like to do RSS on the inner packet fields instead on the outer. When MLX5_RX_HASH_INNER is set with one or more of the other hash fields, then the RSS will be done using the inner packet. Differential revision: https://reviews.freebsd.org/D32196 MFC after: 1 month --- sys/dev/mlx5/mlx5_ib/mlx5_ib.h | 2 ++ sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c | 14 ++++++++++ sys/dev/mlx5/mlx5_ib/mlx5_ib_qp.c | 49 +++++++++++++++++++++++++++++++---- sys/dev/mlx5/mlx5_ifc.h | 2 +- sys/ofed/include/uapi/rdma/mlx5-abi.h | 17 ++++++++++-- 5 files changed, 76 insertions(+), 8 deletions(-) diff --git a/sys/dev/mlx5/mlx5_ib/mlx5_ib.h b/sys/dev/mlx5/mlx5_ib/mlx5_ib.h index 7deb7e1fcbab..6ba8ce659988 100644 --- a/sys/dev/mlx5/mlx5_ib/mlx5_ib.h +++ b/sys/dev/mlx5/mlx5_ib/mlx5_ib.h @@ -445,6 +445,7 @@ struct mlx5_ib_qp { struct list_head qps_list; struct list_head cq_recv_list; struct list_head cq_send_list; + bool tunnel_offload_en; }; struct mlx5_ib_cq_buf { @@ -467,6 +468,7 @@ enum mlx5_ib_qp_flags { MLX5_IB_QP_RSS = 1 << 8, MLX5_IB_QP_CVLAN_STRIPPING = 1 << 9, MLX5_IB_QP_UNDERLAY = 1 << 10, + MLX5_IB_QP_TUNNEL_OFFLOAD = 1 << 12, }; struct mlx5_umr_wr { diff --git a/sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c b/sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c index 8cb7c0bf22d2..7ad96de7f02c 100644 --- a/sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c +++ b/sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c @@ -921,6 +921,20 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, } } + if (field_avail(typeof(resp), tunnel_offloads_caps, + uhw->outlen)) { + resp.response_length += sizeof(resp.tunnel_offloads_caps); + if (MLX5_CAP_ETH(mdev, tunnel_stateless_vxlan)) + resp.tunnel_offloads_caps |= + MLX5_IB_TUNNELED_OFFLOADS_VXLAN; + if (MLX5_CAP_ETH(mdev, tunnel_stateless_geneve_rx)) + resp.tunnel_offloads_caps |= + MLX5_IB_TUNNELED_OFFLOADS_GENEVE; + if (MLX5_CAP_ETH(mdev, tunnel_stateless_gre)) + resp.tunnel_offloads_caps |= + MLX5_IB_TUNNELED_OFFLOADS_GRE; + } + if (uhw->outlen) { err = ib_copy_to_udata(uhw, &resp, resp.response_length); diff --git a/sys/dev/mlx5/mlx5_ib/mlx5_ib_qp.c b/sys/dev/mlx5/mlx5_ib/mlx5_ib_qp.c index 41d3251c8b63..10b427c9c42a 100644 --- a/sys/dev/mlx5/mlx5_ib/mlx5_ib_qp.c +++ b/sys/dev/mlx5/mlx5_ib/mlx5_ib_qp.c @@ -1240,9 +1240,16 @@ static void destroy_raw_packet_qp_rq(struct mlx5_ib_dev *dev, mlx5_core_destroy_rq_tracked(dev->mdev, &rq->base.mqp); } +static bool tunnel_offload_supported(struct mlx5_core_dev *dev) +{ + return (MLX5_CAP_ETH(dev, tunnel_stateless_vxlan) || + MLX5_CAP_ETH(dev, tunnel_stateless_gre) || + MLX5_CAP_ETH(dev, tunnel_stateless_geneve_rx)); +} + static int create_raw_packet_qp_tir(struct mlx5_ib_dev *dev, struct mlx5_ib_rq *rq, u32 tdn, - struct ib_pd *pd) + struct ib_pd *pd, bool tunnel_offload_en) { u32 *in; void *tirc; @@ -1259,6 +1266,8 @@ static int create_raw_packet_qp_tir(struct mlx5_ib_dev *dev, MLX5_SET(tirc, tirc, disp_type, MLX5_TIRC_DISP_TYPE_DIRECT); MLX5_SET(tirc, tirc, inline_rqn, rq->base.mqp.qpn); MLX5_SET(tirc, tirc, transport_domain, tdn); + if (tunnel_offload_en) + MLX5_SET(tirc, tirc, tunneled_offload_en, 1); err = mlx5_core_create_tir(dev->mdev, in, inlen, &rq->tirn); @@ -1309,7 +1318,8 @@ static int create_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp, goto err_destroy_sq; - err = create_raw_packet_qp_tir(dev, rq, tdn, pd); + err = create_raw_packet_qp_tir(dev, rq, tdn, pd, + qp->tunnel_offload_en); if (err) goto err_destroy_rq; } @@ -1397,7 +1407,7 @@ static int create_rss_raw_qp_tir(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp, if (udata->outlen < min_resp_len) return -EINVAL; - required_cmd_sz = offsetof(typeof(ucmd), reserved1) + sizeof(ucmd.reserved1); + required_cmd_sz = offsetof(typeof(ucmd), flags) + sizeof(ucmd.flags); if (udata->inlen < required_cmd_sz) { mlx5_ib_dbg(dev, "invalid inlen\n"); return -EINVAL; @@ -1420,8 +1430,20 @@ static int create_rss_raw_qp_tir(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp, return -EOPNOTSUPP; } - if (memchr_inv(ucmd.reserved, 0, sizeof(ucmd.reserved)) || ucmd.reserved1) { - mlx5_ib_dbg(dev, "invalid reserved\n"); + if (ucmd.flags & ~MLX5_QP_FLAG_TUNNEL_OFFLOADS) { + mlx5_ib_dbg(dev, "invalid flags\n"); + return -EOPNOTSUPP; + } + + if (ucmd.flags & MLX5_QP_FLAG_TUNNEL_OFFLOADS && + !tunnel_offload_supported(dev->mdev)) { + mlx5_ib_dbg(dev, "tunnel offloads isn't supported\n"); + return -EOPNOTSUPP; + } + + if (ucmd.rx_hash_fields_mask & MLX5_RX_HASH_INNER && + !(ucmd.flags & MLX5_QP_FLAG_TUNNEL_OFFLOADS)) { + mlx5_ib_dbg(dev, "Tunnel offloads must be set for inner RSS\n"); return -EOPNOTSUPP; } @@ -1445,6 +1467,15 @@ static int create_rss_raw_qp_tir(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp, MLX5_SET(tirc, tirc, transport_domain, tdn); hfso = MLX5_ADDR_OF(tirc, tirc, rx_hash_field_selector_outer); + + if (ucmd.flags & MLX5_QP_FLAG_TUNNEL_OFFLOADS) + MLX5_SET(tirc, tirc, tunneled_offload_en, 1); + + if (ucmd.rx_hash_fields_mask & MLX5_RX_HASH_INNER) + hfso = MLX5_ADDR_OF(tirc, tirc, rx_hash_field_selector_inner); + else + hfso = MLX5_ADDR_OF(tirc, tirc, rx_hash_field_selector_outer); + switch (ucmd.rx_hash_function) { case MLX5_RX_HASH_FUNC_TOEPLITZ: { @@ -1698,6 +1729,14 @@ static int create_qp_common(struct mlx5_ib_dev *dev, struct ib_pd *pd, qp->wq_sig = !!(ucmd.flags & MLX5_QP_FLAG_SIGNATURE); qp->scat_cqe = !!(ucmd.flags & MLX5_QP_FLAG_SCATTER_CQE); + if (ucmd.flags & MLX5_QP_FLAG_TUNNEL_OFFLOADS) { + if (init_attr->qp_type != IB_QPT_RAW_PACKET || + !tunnel_offload_supported(mdev)) { + mlx5_ib_dbg(dev, "Tunnel offload isn't supported\n"); + return -EOPNOTSUPP; + } + qp->tunnel_offload_en = true; + } } else { qp->wq_sig = !!wq_signature; } diff --git a/sys/dev/mlx5/mlx5_ifc.h b/sys/dev/mlx5/mlx5_ifc.h index a82d8543902d..f2ae0dbfd0b3 100644 --- a/sys/dev/mlx5/mlx5_ifc.h +++ b/sys/dev/mlx5/mlx5_ifc.h @@ -1166,7 +1166,7 @@ struct mlx5_ifc_per_protocol_networking_offload_caps_bits { u8 tunnel_lso_const_out_ip_id[0x1]; u8 tunnel_lro_gre[0x1]; u8 tunnel_lro_vxlan[0x1]; - u8 tunnel_statless_gre[0x1]; + u8 tunnel_stateless_gre[0x1]; u8 tunnel_stateless_vxlan[0x1]; u8 swp[0x1]; diff --git a/sys/ofed/include/uapi/rdma/mlx5-abi.h b/sys/ofed/include/uapi/rdma/mlx5-abi.h index fbd4ce8d86b0..a49eef113688 100644 --- a/sys/ofed/include/uapi/rdma/mlx5-abi.h +++ b/sys/ofed/include/uapi/rdma/mlx5-abi.h @@ -44,6 +44,7 @@ enum { MLX5_QP_FLAG_SIGNATURE = 1 << 0, MLX5_QP_FLAG_SCATTER_CQE = 1 << 1, + MLX5_QP_FLAG_TUNNEL_OFFLOADS = 1 << 2, MLX5_QP_FLAG_BFREG_INDEX = 1 << 3, MLX5_QP_FLAG_UAR_PAGE_INDEX = 1 << 10, }; @@ -197,6 +198,14 @@ struct mlx5_ib_striding_rq_caps { __u32 supported_qpts; }; +enum mlx5_ib_tunnel_offloads { + MLX5_IB_TUNNELED_OFFLOADS_VXLAN = 1 << 0, + MLX5_IB_TUNNELED_OFFLOADS_GRE = 1 << 1, + MLX5_IB_TUNNELED_OFFLOADS_GENEVE = 1 << 2, + MLX5_IB_TUNNELED_OFFLOADS_MPLS_GRE = 1 << 3, + MLX5_IB_TUNNELED_OFFLOADS_MPLS_UDP = 1 << 4, +}; + struct mlx5_ib_query_device_resp { __u32 comp_mask; __u32 response_length; @@ -208,6 +217,8 @@ struct mlx5_ib_query_device_resp { __u32 reserved; struct mlx5_ib_sw_parsing_caps sw_parsing_caps; struct mlx5_ib_striding_rq_caps striding_rq_caps; + __u32 tunnel_offloads_caps; /* enum mlx5_ib_tunnel_offloads */ + __u32 reserved3; }; enum mlx5_ib_create_cq_flags { @@ -286,7 +297,9 @@ enum mlx5_rx_hash_fields { MLX5_RX_HASH_SRC_PORT_TCP = 1 << 4, MLX5_RX_HASH_DST_PORT_TCP = 1 << 5, MLX5_RX_HASH_SRC_PORT_UDP = 1 << 6, - MLX5_RX_HASH_DST_PORT_UDP = 1 << 7 + MLX5_RX_HASH_DST_PORT_UDP = 1 << 7, + /* Save bits for future fields */ + MLX5_RX_HASH_INNER = 1 << 31 }; struct mlx5_ib_create_qp_rss { @@ -296,7 +309,7 @@ struct mlx5_ib_create_qp_rss { __u8 reserved[6]; __u8 rx_hash_key[128]; /* valid only for Toeplitz */ __u32 comp_mask; - __u32 reserved1; + __u32 flags; }; struct mlx5_ib_create_qp_resp {