From nobody Tue Feb 08 15:13:46 2022 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6464F19C3082; Tue, 8 Feb 2022 15:13:48 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JtRNq6Z59z4dgR; Tue, 8 Feb 2022 15:13:47 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1644333228; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=fcXarfCW37qKwKiF6UK4VCjnGbC0w8NVt3gnsi86FuA=; b=RIwaj83498UsuF1YzbZNIUgwBucUKaOmDOhSCbyRbAnzEi5UuJFw1CADgsJ3mDVeLdsf/X Qya0ozv/QnmFHN9gP1jKR7k08GhVppJCrNO8eDXV7JOCg8dcRcoDO9MYaObbVrtJF59QOI MkX3+o2xv+eKcTSk5Jze7EOlk+J6UXGyVtMW+Pss6YkGi1Mpfk1hTHWbmPisEfkdJH0A1N bwHtXwF7nqhSeh7bpR0yaKJgxNsXBj4talOXhyRzbTzrZtyk9aHxtLbO00gBr0RloGtWRp KKSf5ZKayfG32uWfiELRFz0oB6wxeZ1zX2x98NsKt20CcuFLRzTCUiGfGCAZtw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id F269C1BB8D; Tue, 8 Feb 2022 15:13:46 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 218FDk6o020408; Tue, 8 Feb 2022 15:13:46 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 218FDke9020407; Tue, 8 Feb 2022 15:13:46 GMT (envelope-from git) Date: Tue, 8 Feb 2022 15:13:46 GMT Message-Id: <202202081513.218FDke9020407@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Hans Petter Selasky Subject: git: 62283a6345f8 - stable/13 - mlx5en: Implement dummy receive queue, RQ, for dropping packets. List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: hselasky X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 62283a6345f8884a67dd22aa7949b45f1e988a79 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1644333228; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=fcXarfCW37qKwKiF6UK4VCjnGbC0w8NVt3gnsi86FuA=; b=lrtBgeY9/XimXUNe1VOITpYw2Bj/wapvoNKuS6GB/XV181CAZB0F6tGG7YbWO0i+B8jzKq IBABi5RO6lDb3EMD+zO8TGcc8CdTH6Icx4GV0IzuHYV8SbW+m12egiNYag0Q7rupEllBdM /HOuQxKbwPpPrNAT41Dav48E8zmTfKFnX98EMA5pKJitxxjy4pfTaz7DS5cwM+0DZvR1Sb G5ZkMCqiXykQbQNB8JtjKOhDTZEYwfrtConf+n8wXFM6+L6BIFIAaMuplNpwNT+pAEY88H tdkUD+ezl/uxMr2yPFbvjSJqKcewEhrV4uQVo9ktdTHestMsYAe8C5KSZiD5ig== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1644333228; a=rsa-sha256; cv=none; b=xYXHEo4J71hmdTa/K4ImLm8y2mFchrpGRF7nZ5eK9crHmQb9do2rDx70LQD/p4N0jDNIk0 eWvdyTE+12UN/gsNdyj2SF6QbE3V838TiiKkM+TkvFz8HRAE0JTvnPLfffjZ43D+4odjzT gdLSZ2Fy3fGIp2fEPjw4DuKQg1kt2ml+eDcWGQmlx9OAz/PYlWf2T79WUwdHY2Tnif+gOt KMfl6iHte8Ztu4QX/G1oHlF25FxD7JkA7q5mUocY2xBR59W/560rssjC0qKxLevKp8Qptk SpHmUdNXb72xKczREH30rcHoAtA9fXrT8bOPLldUetfsxnWa9ev0AuGemact1w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by hselasky: URL: https://cgit.FreeBSD.org/src/commit/?id=62283a6345f8884a67dd22aa7949b45f1e988a79 commit 62283a6345f8884a67dd22aa7949b45f1e988a79 Author: Hans Petter Selasky AuthorDate: 2022-02-08 15:08:51 +0000 Commit: Hans Petter Selasky CommitDate: 2022-02-08 15:08:51 +0000 mlx5en: Implement dummy receive queue, RQ, for dropping packets. What is a drop RQ and why is it needed? The RSS indirection table, also called the RQT, selects the destination RQ based on the receive queue number, RQN. The RQT is frequently referred to by flow steering rules to distribute traffic among multiple RQs. The problem is that the RQs cannot be destroyed before the RQT referring them is destroyed too. Further, TLS RX rules may still be referring to the RQT even if the link went down. Because there is no magic RQN for dropping packets, we create a dummy RQ, also called drop RQ, which sole purpose is to drop all received packets. When the link goes down this RQN is filled in all RQT entries, of the main RQT, so the real RQs which are about to be destroyed can be released and the TLS RX rules can be sustained. Sponsored by: NVIDIA Networking (cherry picked from commit 27b778ae55374a195ebb889f40200cd5986e9c15) --- sys/dev/mlx5/mlx5_en/en.h | 1 + sys/dev/mlx5/mlx5_en/mlx5_en_main.c | 92 +++++++++++++++++++++++++++++++++++++ 2 files changed, 93 insertions(+) diff --git a/sys/dev/mlx5/mlx5_en/en.h b/sys/dev/mlx5/mlx5_en/en.h index 0265683e090f..9db48c435e8a 100644 --- a/sys/dev/mlx5/mlx5_en/en.h +++ b/sys/dev/mlx5/mlx5_en/en.h @@ -1018,6 +1018,7 @@ struct mlx5e_priv { #define PRIV_LOCKED(priv) sx_xlocked(&(priv)->state_lock) #define PRIV_ASSERT_LOCKED(priv) sx_assert(&(priv)->state_lock, SA_XLOCKED) struct sx state_lock; /* Protects Interface state */ + struct mlx5e_rq drop_rq; u32 pdn; u32 tdn; struct mlx5_core_mr mr; diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c index 126884210b58..13e104c8d2c9 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c @@ -1471,6 +1471,88 @@ mlx5e_close_rq_wait(struct mlx5e_rq *rq) mlx5e_destroy_rq(rq); } +/* + * What is a drop RQ and why is it needed? + * + * The RSS indirection table, also called the RQT, selects the + * destination RQ based on the receive queue number, RQN. The RQT is + * frequently referred to by flow steering rules to distribute traffic + * among multiple RQs. The problem is that the RQs cannot be destroyed + * before the RQT referring them is destroyed too. Further, TLS RX + * rules may still be referring to the RQT even if the link went + * down. Because there is no magic RQN for dropping packets, we create + * a dummy RQ, also called drop RQ, which sole purpose is to drop all + * received packets. When the link goes down this RQN is filled in all + * RQT entries, of the main RQT, so the real RQs which are about to be + * destroyed can be released and the TLS RX rules can be sustained. + */ +static void +mlx5e_open_drop_rq_comp(struct mlx5_core_cq *mcq __unused, struct mlx5_eqe *eqe __unused) +{ +} + +static int +mlx5e_open_drop_rq(struct mlx5e_priv *priv, + struct mlx5e_rq *drop_rq) +{ + struct mlx5e_cq_param param_cq = {}; + struct mlx5e_rq_param param_rq = {}; + void *rqc_wq = MLX5_ADDR_OF(rqc, param_rq.rqc, wq); + int err; + + /* set basic CQ parameters needed */ + MLX5_SET(cqc, param_cq.cqc, log_cq_size, 0); + MLX5_SET(cqc, param_cq.cqc, uar_page, priv->mdev->priv.uar->index); + + /* open receive completion queue */ + err = mlx5e_open_cq(priv, ¶m_cq, &drop_rq->cq, + &mlx5e_open_drop_rq_comp, 0); + if (err) + goto err_done; + + /* set basic WQ parameters needed */ + MLX5_SET(wq, rqc_wq, wq_type, MLX5_WQ_TYPE_LINKED_LIST); + MLX5_SET(wq, rqc_wq, end_padding_mode, MLX5_WQ_END_PAD_MODE_ALIGN); + MLX5_SET(wq, rqc_wq, log_wq_stride, ilog2(sizeof(struct mlx5e_rx_wqe) + sizeof(struct mlx5_wqe_data_seg))); + MLX5_SET(wq, rqc_wq, log_wq_sz, 0); + MLX5_SET(wq, rqc_wq, pd, priv->pdn); + + param_rq.wq.linear = 1; + + err = mlx5_wq_ll_create(priv->mdev, ¶m_rq.wq, rqc_wq, &drop_rq->wq, + &drop_rq->wq_ctrl); + if (err) + goto err_close_cq; + + err = mlx5e_enable_rq(drop_rq, ¶m_rq); + if (err) + goto err_wq_destroy; + + err = mlx5e_modify_rq(drop_rq, MLX5_RQC_STATE_RST, MLX5_RQC_STATE_RDY); + if (err) + goto err_disable_rq; + + return (err); + +err_disable_rq: + mlx5e_disable_rq(drop_rq); +err_wq_destroy: + mlx5_wq_destroy(&drop_rq->wq_ctrl); +err_close_cq: + mlx5e_close_cq(&drop_rq->cq); +err_done: + return (err); +} + +static void +mlx5e_close_drop_rq(struct mlx5e_rq *drop_rq) +{ + mlx5e_modify_rq(drop_rq, MLX5_RQC_STATE_RDY, MLX5_RQC_STATE_ERR); + mlx5e_disable_rq(drop_rq); + mlx5_wq_destroy(&drop_rq->wq_ctrl); + mlx5e_close_cq(&drop_rq->cq); +} + void mlx5e_free_sq_db(struct mlx5e_sq *sq) { @@ -4519,6 +4601,12 @@ mlx5e_create_ifp(struct mlx5_core_dev *mdev) goto err_rl_init; } + err = mlx5e_open_drop_rq(priv, &priv->drop_rq); + if (err) { + if_printf(ifp, "%s: mlx5e_open_drop_rq failed\n", __func__); + goto err_tls_init; + } + /* set default MTU */ mlx5e_set_dev_port_mtu(ifp, ifp->if_mtu); @@ -4646,6 +4734,9 @@ mlx5e_create_ifp(struct mlx5_core_dev *mdev) return (priv); +err_tls_init: + mlx5e_tls_cleanup(priv); + err_rl_init: mlx5e_rl_cleanup(priv); @@ -4741,6 +4832,7 @@ mlx5e_destroy_ifp(struct mlx5_core_dev *mdev, void *vpriv) ifmedia_removeall(&priv->media); ether_ifdetach(ifp); + mlx5e_close_drop_rq(&priv->drop_rq); mlx5e_tls_cleanup(priv); mlx5e_rl_cleanup(priv);