From nobody Tue Feb 08 15:14:13 2022 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id CB7D719C413B; Tue, 8 Feb 2022 15:14:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JtRPK50RFz4dqV; Tue, 8 Feb 2022 15:14:13 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1644333255; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=dW2fWcTSZuUHiZcksh2jpb67+nayQq7+PfhbolTse0M=; b=hsg+E+WcKgzC6Vhh2DlL0+eXD2jlaU/zUVm2Ao8jPJjf7itTtlf8zzOk9CyH/81c6aMcJD LZ/1XinO5OzETx+upuJW4zVxc1DqNgWlCQJ2MwsrHV9OSmIbxwTyPQK6Quke7lIJYIHosy ocFZ6QSCXTBOitz9COx9JEV0npqFZfgsqQv5c6ffWbjYXUXlHMErroGVI3JkS2wMSgr3+K UcL/IB2d1/o99fykHkhogOidKuKYM/lkDqi4m064tqu4+ycY4KCONX3RIRQQdABXerjr2x cO/g63n8VXgfOc2154nHQ+KcOZHuhj8hJlgF8iBZI5FMu7PseqVE3o4usicQrg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 2D2721BB97; Tue, 8 Feb 2022 15:14:13 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 218FEDKb020992; Tue, 8 Feb 2022 15:14:13 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 218FEDYh020991; Tue, 8 Feb 2022 15:14:13 GMT (envelope-from git) Date: Tue, 8 Feb 2022 15:14:13 GMT Message-Id: <202202081514.218FEDYh020991@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Hans Petter Selasky Subject: git: 823bcb3a13f0 - stable/13 - mlx5: Add raw ethernet local loopback support. List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: hselasky X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 823bcb3a13f0f2ee04fd2d4addbbfde479e10e48 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1644333255; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=dW2fWcTSZuUHiZcksh2jpb67+nayQq7+PfhbolTse0M=; b=hJHC0h1teCGp100lFRx8hQcZy2diVFm/YM0ABxqPGI6Yo7vzYECPpDEnZ+RT9+rjlRGU8t MUBn2LnLnphDAHcWibNXF/X0s11XAwA/p/0BIY5Oqop6O8zxkOWnIiKa+mgeEVyFh1RFG2 wsxFO0eaBFef9hos6ynPC7kKePeX6fU2iWkDEAXf6G+6GSWs4afKXDei22uDfXX6qrCHJP bg2osUMy9MtdqMU1b04C/Z6FTKgn4jB62PLNUHjqjxYOzq2nE77XORPlfNr5WOiy6QimzD jVf2LoBruX0qjIuycUBRQuM5YRkySmhdHuwZ94cmNtnBkG7NehbjFOPSmd3Q5Q== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1644333255; a=rsa-sha256; cv=none; b=cuW5Z7qFg7Z/vdHmzO1LerEtFSnWeXFvD4TiDOEorQG8SDucMrzAtxH+5K2khYrzv/5KCg Hk0Q7/I1aZ8zMOefGC1odqQXyYWpOi63ItAqaR+4dnkTh/ObPvfLg8h/whQkm6oFSw9UCy MJ+azX6gqB/RNCVG+jjwpdBuoHFfejsMv7IYk3FEYzomisSVWQXQ4MvavmhZSYWYesNp2D uBkTdezXvTiN8UcsrQnCxbFfHKDlCF3Jvj/L7kr8H3gjINQ7RIxiScj6oLAfntrJ/iouf0 khOy1mszKeZcihFm6VJp2GQr0YFfvspR2xhkE0uwvx1/40sVKmsifFvZ1FZAWg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by hselasky: URL: https://cgit.FreeBSD.org/src/commit/?id=823bcb3a13f0f2ee04fd2d4addbbfde479e10e48 commit 823bcb3a13f0f2ee04fd2d4addbbfde479e10e48 Author: Hans Petter Selasky AuthorDate: 2022-02-08 15:08:54 +0000 Commit: Hans Petter Selasky CommitDate: 2022-02-08 15:08:54 +0000 mlx5: Add raw ethernet local loopback support. Currently, unicast/multicast loopback raw ethernet (non-RDMA) packets are sent back to the vport. A unicast loopback packet is the packet with destination MAC address the same as the source MAC address. For multicast, the destination MAC address is in the vport's multicast filter list. Moreover, the local loopback is not needed if there is one or none user space context. After this patch, the raw ethernet unicast and multicast local loopback are disabled by default. When there is more than one user space context, the local loopback is enabled. Note that when local loopback is disabled, raw ethernet packets are not looped back to the vport and are forwarded to the next routing level (eswitch, or multihost switch, or out to the wire depending on the configuration). Linux commits: c85023e153e3824661d07307138fdeff41f6d86a 8978cc921fc7fad3f4d6f91f1da01352aeeeff25 Sponsored by: NVIDIA Networking (cherry picked from commit ea00d7e8cab97f818c9793a112249ab79a2f8322) --- sys/dev/mlx5/mlx5_core/mlx5_main.c | 17 +++++++++ sys/dev/mlx5/mlx5_en/mlx5_en_ethtool.c | 12 ++++++ sys/dev/mlx5/mlx5_ib/mlx5_ib.h | 4 ++ sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c | 67 +++++++++++++++++++++++++++++----- 4 files changed, 90 insertions(+), 10 deletions(-) diff --git a/sys/dev/mlx5/mlx5_core/mlx5_main.c b/sys/dev/mlx5/mlx5_core/mlx5_main.c index 0745d19b2725..aaa8f657432a 100644 --- a/sys/dev/mlx5/mlx5_core/mlx5_main.c +++ b/sys/dev/mlx5/mlx5_core/mlx5_main.c @@ -532,6 +532,17 @@ static int set_hca_ctrl(struct mlx5_core_dev *dev) return err; } +static int mlx5_core_set_hca_defaults(struct mlx5_core_dev *dev) +{ + int ret = 0; + + /* Disable local_lb by default */ + if (MLX5_CAP_GEN(dev, port_type) == MLX5_CAP_PORT_TYPE_ETH) + ret = mlx5_nic_vport_update_local_lb(dev, false); + + return ret; +} + static int mlx5_core_enable_hca(struct mlx5_core_dev *dev, u16 func_id) { u32 out[MLX5_ST_SZ_DW(enable_hca_out)] = {0}; @@ -1135,6 +1146,12 @@ static int mlx5_load_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv, goto err_free_comp_eqs; } + err = mlx5_core_set_hca_defaults(dev); + if (err) { + mlx5_core_err(dev, "Failed to set HCA defaults %d\n", err); + goto err_free_comp_eqs; + } + err = mlx5_mpfs_init(dev); if (err) { mlx5_core_err(dev, "mpfs init failed %d\n", err); diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_ethtool.c b/sys/dev/mlx5/mlx5_en/mlx5_en_ethtool.c index d07aee1f0793..f57d70080a46 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_ethtool.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_ethtool.c @@ -1230,6 +1230,12 @@ mlx5e_ethtool_handler(SYSCTL_HANDLER_ARGS) break; case MLX5_PARAM_OFFSET(mc_local_lb): + /* check if mlx5ib is managing this feature */ + if (MLX5_CAP_GEN(priv->mdev, port_type) != MLX5_CAP_PORT_TYPE_ETH) { + error = EOPNOTSUPP; + break; + } + priv->params_ethtool.mc_local_lb = priv->params_ethtool.mc_local_lb ? 1 : 0; @@ -1242,6 +1248,12 @@ mlx5e_ethtool_handler(SYSCTL_HANDLER_ARGS) break; case MLX5_PARAM_OFFSET(uc_local_lb): + /* check if mlx5ib is managing this feature */ + if (MLX5_CAP_GEN(priv->mdev, port_type) != MLX5_CAP_PORT_TYPE_ETH) { + error = EOPNOTSUPP; + break; + } + priv->params_ethtool.uc_local_lb = priv->params_ethtool.uc_local_lb ? 1 : 0; diff --git a/sys/dev/mlx5/mlx5_ib/mlx5_ib.h b/sys/dev/mlx5/mlx5_ib/mlx5_ib.h index 49f6e87868ff..47f9cc12d51a 100644 --- a/sys/dev/mlx5/mlx5_ib/mlx5_ib.h +++ b/sys/dev/mlx5/mlx5_ib/mlx5_ib.h @@ -769,6 +769,10 @@ struct mlx5_ib_dev { struct mlx5_ib_congestion congestion; struct mlx5_async_ctx async_ctx; + + /* protect the user_td */ + struct mutex lb_mutex; + u32 user_td; }; static inline struct mlx5_ib_cq *to_mibcq(struct mlx5_core_cq *mcq) diff --git a/sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c b/sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c index f517dfc58fac..95214f6003ce 100644 --- a/sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c +++ b/sys/dev/mlx5/mlx5_ib/mlx5_ib_main.c @@ -1187,6 +1187,56 @@ static void deallocate_uars(struct mlx5_ib_dev *dev, mlx5_cmd_free_uar(dev->mdev, bfregi->sys_pages[i]); } +static int mlx5_ib_alloc_transport_domain(struct mlx5_ib_dev *dev, u32 *tdn) +{ + int err; + + if (!MLX5_CAP_GEN(dev->mdev, log_max_transport_domain)) + return 0; + + err = mlx5_alloc_transport_domain(dev->mdev, tdn); + if (err) + return err; + + if ((MLX5_CAP_GEN(dev->mdev, port_type) != MLX5_CAP_PORT_TYPE_ETH) || + (!MLX5_CAP_GEN(dev->mdev, disable_local_lb_uc) && + !MLX5_CAP_GEN(dev->mdev, disable_local_lb_mc))) + return 0; + + mutex_lock(&dev->lb_mutex); + dev->user_td++; + + if (dev->user_td == 2) + err = mlx5_nic_vport_update_local_lb(dev->mdev, true); + + mutex_unlock(&dev->lb_mutex); + + if (err != 0) + mlx5_dealloc_transport_domain(dev->mdev, *tdn); + return err; +} + +static void mlx5_ib_dealloc_transport_domain(struct mlx5_ib_dev *dev, u32 tdn) +{ + if (!MLX5_CAP_GEN(dev->mdev, log_max_transport_domain)) + return; + + mlx5_dealloc_transport_domain(dev->mdev, tdn); + + if ((MLX5_CAP_GEN(dev->mdev, port_type) != MLX5_CAP_PORT_TYPE_ETH) || + (!MLX5_CAP_GEN(dev->mdev, disable_local_lb_uc) && + !MLX5_CAP_GEN(dev->mdev, disable_local_lb_mc))) + return; + + mutex_lock(&dev->lb_mutex); + dev->user_td--; + + if (dev->user_td < 2) + mlx5_nic_vport_update_local_lb(dev->mdev, false); + + mutex_unlock(&dev->lb_mutex); +} + static struct ib_ucontext *mlx5_ib_alloc_ucontext(struct ib_device *ibdev, struct ib_udata *udata) { @@ -1301,12 +1351,9 @@ uar_done: context->ibucontext.invalidate_range = &mlx5_ib_invalidate_range; #endif - if (MLX5_CAP_GEN(dev->mdev, log_max_transport_domain)) { - err = mlx5_alloc_transport_domain(dev->mdev, - &context->tdn); - if (err) - goto out_uars; - } + err = mlx5_ib_alloc_transport_domain(dev, &context->tdn); + if (err) + goto out_uars; INIT_LIST_HEAD(&context->vma_private_list); INIT_LIST_HEAD(&context->db_page_list); @@ -1362,8 +1409,7 @@ uar_done: return &context->ibucontext; out_td: - if (MLX5_CAP_GEN(dev->mdev, log_max_transport_domain)) - mlx5_dealloc_transport_domain(dev->mdev, context->tdn); + mlx5_ib_dealloc_transport_domain(dev, context->tdn); out_uars: deallocate_uars(dev, context); @@ -1386,8 +1432,7 @@ static int mlx5_ib_dealloc_ucontext(struct ib_ucontext *ibcontext) struct mlx5_bfreg_info *bfregi; bfregi = &context->bfregi; - if (MLX5_CAP_GEN(dev->mdev, log_max_transport_domain)) - mlx5_dealloc_transport_domain(dev->mdev, context->tdn); + mlx5_ib_dealloc_transport_domain(dev, context->tdn); deallocate_uars(dev, context); kfree(bfregi->sys_pages); @@ -3289,6 +3334,8 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev) MLX5_INIT_DOORBELL_LOCK(&dev->uar_lock); + mutex_init(&dev->lb_mutex); + snprintf(dev->ib_dev.name, IB_DEVICE_NAME_MAX, "mlx5_%d", device_get_unit(mdev->pdev->dev.bsddev)); dev->ib_dev.owner = THIS_MODULE; dev->ib_dev.node_type = RDMA_NODE_IB_CA;