From nobody Tue Feb 01 15:24:14 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id F11BF1915605; Tue, 1 Feb 2022 15:24:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jp7y71zHyz3FqW; Tue, 1 Feb 2022 15:24:15 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1643729055; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=JiR7QRoH8OyzQyAjMRl4l3pSVrEf3ZJu0OkEj9M7Svk=; b=cOkb3wYH5RmNDz8Lli4ggK431nEiL/3AG9s/buAUtDg8ZrrXY53AOFP1L03PNSz97shFl2 amLi0yze+sBJE4yIcjYpZVzDi+9YiYkaKpR6gDV3bt7caP688iMsR2D3t66dVrz9KIvz7C 3eAlQGJnMEGO9+HMEDkMbckzMiqJ6rFawjEOT5SmJegj/G5EzorXxpON+xRIGdjpZMnB41 oJ+y1+XYztaTHyaVrDP8Hv1hSVS5Sw7i2vbtKo0u2S2sa+uuLhBrpJto9CoH6fmsn5atGN 5f5W3VA4rXJBguu+SAlQY1Fgp90mN5BUaOrWV7UKIy8l4r4OxJk23M38jpTrzQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 83CF7133B8; Tue, 1 Feb 2022 15:24:14 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 211FOEAb050745; Tue, 1 Feb 2022 15:24:14 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 211FOEIu050744; Tue, 1 Feb 2022 15:24:14 GMT (envelope-from git) Date: Tue, 1 Feb 2022 15:24:14 GMT Message-Id: <202202011524.211FOEIu050744@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Hans Petter Selasky Subject: git: 33a6a7a72abc - main - mlx5en: Make the receive packet indirection table, RQT, static instead of dynamic. List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: hselasky X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 33a6a7a72abc1e1c9ecb4b3d12087731fd11f0bb Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1643729055; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=JiR7QRoH8OyzQyAjMRl4l3pSVrEf3ZJu0OkEj9M7Svk=; b=lDr7oVmUAuXsXpAHFuZg4ly00opn9DA+OQg2KvsUmCxDZQbK9f/hOHoA1AIbln/6Fy4XS9 xVW778w419sam40HgMoCaBLeEYfpNCdTbIyQ9MWFNNNMmb3KIQ8PPanBJjmx6TxWxlPxl6 jNiPDlUXD5ZXI8wl8927hUokhJ4WTcKbvOSoiuV7eqXRjtrTPPMMuW1x33OATzcN+xhZ2O Qu4llQA+Czq+YJ7jppkcV1oZH5keyyPjD8jXVBOUK7USfiCc2YwgBpIhttR4qzYED1ORhy xYAT1wbbdI0iCRl+KHtNI4giioHPp/ciFmwH+l0czzZncAKmQiVmjx9+rUmQoA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1643729055; a=rsa-sha256; cv=none; b=fB3na2UC1oovPp3qmVrD1avsS6MXNRMkAolBl+IReXuvszZSEG2uPE3CPFHAtac04WgHym lwwkA8xOBynPavinGipzvjJInd8vFyJTtknj5qyILWZmTQLSa5DQK+2nU/XYF3s6d5MLoV LCscoyf8S7HOrFDJoRBfEDJ+0H4lUNbVdONXcimS5hwdU2II/QqDoBWKE5tzzJcke3K7Bw /EQpR9klm4sXnw5uxHsNJ+AxfdEAiVdtkMQJsygpFvSkyq6BXsdkySV6aCDs6SzbqemHLp k90PZfSGofrv+PPj4siHYamQdg/vsuZCnLNAlUNaUVu7YVOiXufRA6gTnyvlog== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by hselasky: URL: https://cgit.FreeBSD.org/src/commit/?id=33a6a7a72abc1e1c9ecb4b3d12087731fd11f0bb commit 33a6a7a72abc1e1c9ecb4b3d12087731fd11f0bb Author: Hans Petter Selasky AuthorDate: 2022-02-01 15:20:07 +0000 Commit: Hans Petter Selasky CommitDate: 2022-02-01 15:21:14 +0000 mlx5en: Make the receive packet indirection table, RQT, static instead of dynamic. Allocate the RQT once, pointing all initial entries to the drop RQN. When opening the channels simplify modify the RQT, directing all traffic to the new RQNs. Similarly when closing the channels point all RQT entries back to the so-called drop RQN. MFC after: 1 week Sponsored by: NVIDIA Networking --- sys/dev/mlx5/mlx5_en/mlx5_en_main.c | 93 +++++++++++++++++++++++++++++-------- sys/dev/mlx5/mlx5_ifc.h | 9 +++- 2 files changed, 81 insertions(+), 21 deletions(-) diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c index 8d840d54b964..aec92d4bd08c 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c @@ -2740,9 +2740,7 @@ mlx5e_close_tises(struct mlx5e_priv *priv) static int mlx5e_open_rqt(struct mlx5e_priv *priv) { - struct mlx5_core_dev *mdev = priv->mdev; u32 *in; - u32 out[MLX5_ST_SZ_DW(create_rqt_out)] = {0}; void *rqtc; int inlen; int err; @@ -2760,7 +2758,38 @@ mlx5e_open_rqt(struct mlx5e_priv *priv) MLX5_SET(rqtc, rqtc, rqt_actual_size, sz); MLX5_SET(rqtc, rqtc, rqt_max_size, sz); - for (i = 0; i < sz; i++) { + for (i = 0; i != sz; i++) + MLX5_SET(rqtc, rqtc, rq_num[i], priv->drop_rq.rqn); + + err = mlx5_core_create_rqt(priv->mdev, in, inlen, &priv->rqtn); + kvfree(in); + + return (err); +} + +static int +mlx5e_activate_rqt(struct mlx5e_priv *priv) +{ + u32 *in; + void *rqtc; + int inlen; + int err; + int sz; + int i; + + sz = 1 << priv->params.rx_hash_log_tbl_sz; + + inlen = MLX5_ST_SZ_BYTES(modify_rqt_in) + sizeof(u32) * sz; + in = mlx5_vzalloc(inlen); + if (in == NULL) + return (-ENOMEM); + + rqtc = MLX5_ADDR_OF(modify_rqt_in, in, ctx); + + MLX5_SET(rqtc, rqtc, rqt_actual_size, sz); + MLX5_SET(modify_rqt_in, in, bitmask.rqn_list, 1); + + for (i = 0; i != sz; i++) { int ix = i; #ifdef RSS ix = rss_get_indirection_to_bucket(ix); @@ -2774,27 +2803,41 @@ mlx5e_open_rqt(struct mlx5e_priv *priv) MLX5_SET(rqtc, rqtc, rq_num[i], priv->channel[ix].rq.rqn); } - MLX5_SET(create_rqt_in, in, opcode, MLX5_CMD_OP_CREATE_RQT); - - err = mlx5_cmd_exec(mdev, in, inlen, out, sizeof(out)); - if (!err) - priv->rqtn = MLX5_GET(create_rqt_out, out, rqtn); - + err = mlx5_core_modify_rqt(priv->mdev, priv->rqtn, in, inlen); kvfree(in); return (err); } -static void -mlx5e_close_rqt(struct mlx5e_priv *priv) +static int +mlx5e_deactivate_rqt(struct mlx5e_priv *priv) { - u32 in[MLX5_ST_SZ_DW(destroy_rqt_in)] = {0}; - u32 out[MLX5_ST_SZ_DW(destroy_rqt_out)] = {0}; + u32 *in; + void *rqtc; + int inlen; + int err; + int sz; + int i; + + sz = 1 << priv->params.rx_hash_log_tbl_sz; + + inlen = MLX5_ST_SZ_BYTES(modify_rqt_in) + sizeof(u32) * sz; + in = mlx5_vzalloc(inlen); + if (in == NULL) + return (-ENOMEM); - MLX5_SET(destroy_rqt_in, in, opcode, MLX5_CMD_OP_DESTROY_RQT); - MLX5_SET(destroy_rqt_in, in, rqtn, priv->rqtn); + rqtc = MLX5_ADDR_OF(modify_rqt_in, in, ctx); - mlx5_cmd_exec(priv->mdev, in, sizeof(in), out, sizeof(out)); + MLX5_SET(rqtc, rqtc, rqt_actual_size, sz); + MLX5_SET(modify_rqt_in, in, bitmask.rqn_list, 1); + + for (i = 0; i != sz; i++) + MLX5_SET(rqtc, rqtc, rq_num[i], priv->drop_rq.rqn); + + err = mlx5_core_modify_rqt(priv->mdev, priv->rqtn, in, inlen); + kvfree(in); + + return (err); } #define MLX5E_RSS_KEY_SIZE (10 * 4) /* bytes */ @@ -3164,9 +3207,9 @@ mlx5e_open_locked(struct ifnet *ifp) "mlx5e_open_channels failed, %d\n", err); goto err_dalloc_q_counter; } - err = mlx5e_open_rqt(priv); + err = mlx5e_activate_rqt(priv); if (err) { - mlx5_en_err(ifp, "mlx5e_open_rqt failed, %d\n", err); + mlx5_en_err(ifp, "mlx5e_activate_rqt failed, %d\n", err); goto err_close_channels; } err = mlx5e_open_tirs(priv, false); @@ -3223,7 +3266,7 @@ err_close_tirs: mlx5e_close_tirs(priv, false); err_close_rqls: - mlx5e_close_rqt(priv); + mlx5e_deactivate_rqt(priv); err_close_channels: mlx5e_close_channels(priv); @@ -3273,7 +3316,7 @@ mlx5e_close_locked(struct ifnet *ifp) if ((ifp->if_capenable & IFCAP_VXLAN_HWCSUM) != 0) mlx5e_close_tirs(priv, true); mlx5e_close_tirs(priv, false); - mlx5e_close_rqt(priv); + mlx5e_deactivate_rqt(priv); mlx5e_close_channels(priv); mlx5_vport_dealloc_q_counter(priv->mdev, MLX5_INTERFACE_PROTOCOL_ETH, priv->counter_set_id); @@ -4546,6 +4589,12 @@ mlx5e_create_ifp(struct mlx5_core_dev *mdev) goto err_tls_init; } + err = mlx5e_open_rqt(priv); + if (err) { + if_printf(ifp, "%s: mlx5e_open_rqt failed (%d)\n", __func__, err); + goto err_open_drop_rq; + } + /* set default MTU */ mlx5e_set_dev_port_mtu(ifp, ifp->if_mtu); @@ -4673,6 +4722,9 @@ mlx5e_create_ifp(struct mlx5_core_dev *mdev) return (priv); +err_open_drop_rq: + mlx5e_close_drop_rq(&priv->drop_rq); + err_tls_init: mlx5e_tls_cleanup(priv); @@ -4771,6 +4823,7 @@ mlx5e_destroy_ifp(struct mlx5_core_dev *mdev, void *vpriv) ifmedia_removeall(&priv->media); ether_ifdetach(ifp); + mlx5_core_destroy_rqt(priv->mdev, priv->rqtn, 0); mlx5e_close_drop_rq(&priv->drop_rq); mlx5e_tls_cleanup(priv); mlx5e_rl_cleanup(priv); diff --git a/sys/dev/mlx5/mlx5_ifc.h b/sys/dev/mlx5/mlx5_ifc.h index f80e3e024c9e..27744c2155ea 100644 --- a/sys/dev/mlx5/mlx5_ifc.h +++ b/sys/dev/mlx5/mlx5_ifc.h @@ -5745,6 +5745,13 @@ struct mlx5_ifc_modify_rqt_out_bits { u8 reserved_1[0x40]; }; +struct mlx5_ifc_rqt_bitmask_bits { + u8 reserved_at_0[0x20]; + + u8 reserved_at_20[0x1f]; + u8 rqn_list[0x1]; +}; + struct mlx5_ifc_modify_rqt_in_bits { u8 opcode[0x10]; u8 uid[0x10]; @@ -5757,7 +5764,7 @@ struct mlx5_ifc_modify_rqt_in_bits { u8 reserved_3[0x20]; - u8 modify_bitmask[0x40]; + struct mlx5_ifc_rqt_bitmask_bits bitmask; u8 reserved_4[0x40];