From nobody Tue Feb 08 15:13:43 2022 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 8C8E619C2AE5; Tue, 8 Feb 2022 15:13:44 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JtRNm01XPz4dPP; Tue, 8 Feb 2022 15:13:43 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1644333224; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=yxSwQ1IFp0t+fCddAouJJbN6NIdQO2iXDwQX7hO515s=; b=ARz+NX5chldASl5GlaeYWnBJ6htUBVdYecDSgTmOE252QPFlfX2ikjEFxLKxujB5A0XoBQ lVxnaS0YJCtCwf8Y2gtOOZb72gZ10cZgW4SLY6jFjtvWBwc3a34iqwJruXfpwC4AgJR+Re UZoX62obx9WNESjIJ908n/IAi1HGM17+ie57SoRDf874G5bJH6+gncowZNOcHMxETkyBxn kRuLoaKLkyo+LG2DtmWSw7onjnO1WqkiuD02krrdSIy5q1BhQD3xIDkWOm4+/Scery22XT C4kSaGzGXOrzyII3ufdLXizxMAP+v86bMt+69liE9vJclIjfpPePwkOunzXQXQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9E8AA1BB8C; Tue, 8 Feb 2022 15:13:43 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 218FDhfp020335; Tue, 8 Feb 2022 15:13:43 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 218FDhCx020334; Tue, 8 Feb 2022 15:13:43 GMT (envelope-from git) Date: Tue, 8 Feb 2022 15:13:43 GMT Message-Id: <202202081513.218FDhCx020334@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Hans Petter Selasky Subject: git: 3be2f639a837 - stable/13 - mlx5en: Patch to inhibit transmit doorbell writes during packet reception. List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: hselasky X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 3be2f639a837f1e13de96e20382e3a2539a3f1f8 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1644333224; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=yxSwQ1IFp0t+fCddAouJJbN6NIdQO2iXDwQX7hO515s=; b=CrOnWI6MZd8Ag8bt1Jen4eIjZOz10fh/t7v40ypdHooNxMHLIHhhAkIP8VwgH2QePgIGPG i+zZoEBj4ZpHCn2tpdOdsMOdNqGTqWQTMDwPWHKxHu8A+KnUkdWWfNWa7ZuSdkfqAyA2gs 11pA7rWYe+Otg5HeJ5nWLCGPFJSZi+6KPNuRMHgpmRO3B67ftPw7O+1H2LX0iajBCRzD/+ ULj35wc+HY9Mw5rRu1sCeAGUfek2JhB0zmoN7Lg1BAkRnYYSd5a5PspzveKw78YzM+veOF MjmNkumLhSAihcLIH+KhRouH5Te/VqiDVi8dvBN4kPqRbUN1Xj27QjRh1ZcnGQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1644333224; a=rsa-sha256; cv=none; b=MNgW0s2Zs9xjn6piuat78hu2gylz5VX1zTpmOpwjERz6wJQwrvsJN7Wk1TL9dcpwVyrxLc mQrlouV46uow+J63BaeLJqNyaBWTGziJhG8czV+Oo5o+wyywFgev86aYBKQnbQ41ZnUaX2 VechXZShCiGx+owMEpvHsZiD2lPy/KADsIteIoDnFF++HR5o8thvNyN30ola9VEm0G4nTQ /o3sd3tZuyhgKGby1TaRBdta94S4a1sjEpeUWHJBk5lRr8HW51lw5e+fUuWElCDgNDGmfk YaIUJ8UYeCkTBCkFQqJxJfEe491la7mZndMWTmQrcrYWPTRABZgZW6EqHmsvig== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by hselasky: URL: https://cgit.FreeBSD.org/src/commit/?id=3be2f639a837f1e13de96e20382e3a2539a3f1f8 commit 3be2f639a837f1e13de96e20382e3a2539a3f1f8 Author: Hans Petter Selasky AuthorDate: 2022-02-08 15:08:51 +0000 Commit: Hans Petter Selasky CommitDate: 2022-02-08 15:08:51 +0000 mlx5en: Patch to inhibit transmit doorbell writes during packet reception. During packet reception the network stack frequently transmit data in response to TCP window updates. To reduce the number of transmit doorbells needed, inhibit all transmit doorbells designated for the same channel until after the reception of packets for the given channel is completed. While at it slightly refactor the mlx5e_tx_notify_hw() function: 1) The doorbell information is always stored into sq->doorbell.d64 . No need to pass a separate pointer to this variable. 2) Move checks for skipping doorbell writes inside this function. Sponsored by: NVIDIA Networking (cherry picked from commit 2d5e5a0d75b0612f8cea0f7f59f7d2e59b0c7a60) --- sys/dev/mlx5/mlx5_en/en.h | 12 ++++++++++-- sys/dev/mlx5/mlx5_en/mlx5_en_main.c | 15 ++++----------- sys/dev/mlx5/mlx5_en/mlx5_en_rx.c | 16 +++++++++++++++- sys/dev/mlx5/mlx5_en/mlx5_en_tx.c | 9 +++------ 4 files changed, 32 insertions(+), 20 deletions(-) diff --git a/sys/dev/mlx5/mlx5_en/en.h b/sys/dev/mlx5/mlx5_en/en.h index 773f3f56026a..0265683e090f 100644 --- a/sys/dev/mlx5/mlx5_en/en.h +++ b/sys/dev/mlx5/mlx5_en/en.h @@ -797,6 +797,7 @@ struct mlx5e_sq { struct mtx comp_lock; struct mlx5e_sq_stats stats; struct callout cev_callout; + int db_inhibit; /* data path */ #define mlx5e_sq_zero_start dma_tag @@ -1154,8 +1155,13 @@ int mlx5e_add_all_vxlan_rules(struct mlx5e_priv *priv); void mlx5e_del_all_vxlan_rules(struct mlx5e_priv *priv); static inline void -mlx5e_tx_notify_hw(struct mlx5e_sq *sq, u32 *wqe) +mlx5e_tx_notify_hw(struct mlx5e_sq *sq, bool force) { + if (unlikely((force == false && sq->db_inhibit != 0) || sq->doorbell.d64 == 0)) { + /* skip writing the doorbell record */ + return; + } + /* ensure wqe is visible to device before updating doorbell record */ wmb(); @@ -1167,8 +1173,10 @@ mlx5e_tx_notify_hw(struct mlx5e_sq *sq, u32 *wqe) */ wmb(); - mlx5_write64(wqe, sq->uar_map, + mlx5_write64(sq->doorbell.d32, sq->uar_map, MLX5_GET_DOORBELL_LOCK(&sq->priv->doorbell_lock)); + + sq->doorbell.d64 = 0; } static inline void diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c index 64d8752abad6..4209e3fde72f 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c @@ -1799,11 +1799,7 @@ mlx5e_sq_send_nops_locked(struct mlx5e_sq *sq, int can_sleep) atomic_thread_fence_rel(); } done: - /* Check if we need to write the doorbell */ - if (likely(sq->doorbell.d64 != 0)) { - mlx5e_tx_notify_hw(sq, sq->doorbell.d32); - sq->doorbell.d64 = 0; - } + mlx5e_tx_notify_hw(sq, false); } void @@ -3859,8 +3855,7 @@ mlx5e_reset_sq_doorbell_record(struct mlx5e_sq *sq) sq->doorbell.d32[0] = cpu_to_be32(MLX5_OPCODE_NOP); sq->doorbell.d32[1] = cpu_to_be32(sq->sqn << 8); - mlx5e_tx_notify_hw(sq, sq->doorbell.d32); - sq->doorbell.d64 = 0; + mlx5e_tx_notify_hw(sq, true); } void @@ -4854,10 +4849,8 @@ mlx5_en_debugnet_transmit(struct ifnet *dev, struct mbuf *m) err = 0; } - if (likely(sq->doorbell.d64 != 0)) { - mlx5e_tx_notify_hw(sq, sq->doorbell.d32); - sq->doorbell.d64 = 0; - } + mlx5e_tx_notify_hw(sq, true); + return (err); } diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c b/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c index e943a65709cf..ec6b027b324e 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2015 Mellanox Technologies. All rights reserved. + * Copyright (c) 2015-2021 Mellanox Technologies. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions @@ -566,6 +566,7 @@ wq_ll_pop: void mlx5e_rx_cq_comp(struct mlx5_core_cq *mcq, struct mlx5_eqe *eqe __unused) { + struct mlx5e_channel *c = container_of(mcq, struct mlx5e_channel, rq.cq.mcq); struct mlx5e_rq *rq = container_of(mcq, struct mlx5e_rq, cq.mcq); int i = 0; @@ -584,6 +585,11 @@ mlx5e_rx_cq_comp(struct mlx5_core_cq *mcq, struct mlx5_eqe *eqe __unused) rq->ifp->if_input(rq->ifp, mb); } #endif + for (int j = 0; j != MLX5E_MAX_TX_NUM_TC; j++) { + mtx_lock(&c->sq[j].lock); + c->sq[j].db_inhibit++; + mtx_unlock(&c->sq[j].lock); + } mtx_lock(&rq->mtx); @@ -607,4 +613,12 @@ mlx5e_rx_cq_comp(struct mlx5_core_cq *mcq, struct mlx5_eqe *eqe __unused) mlx5e_cq_arm(&rq->cq, MLX5_GET_DOORBELL_LOCK(&rq->channel->priv->doorbell_lock)); tcp_lro_flush_all(&rq->lro); mtx_unlock(&rq->mtx); + + for (int j = 0; j != MLX5E_MAX_TX_NUM_TC; j++) { + mtx_lock(&c->sq[j].lock); + c->sq[j].db_inhibit--; + /* Update the doorbell record, if any. */ + mlx5e_tx_notify_hw(c->sq + j, true); + mtx_unlock(&c->sq[j].lock); + } } diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c b/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c index e2051896a003..35ab3d016392 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2015-2019 Mellanox Technologies. All rights reserved. + * Copyright (c) 2015-2021 Mellanox Technologies. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions @@ -1122,11 +1122,8 @@ mlx5e_xmit_locked(struct ifnet *ifp, struct mlx5e_sq *sq, struct mbuf *mb) err = ENOBUFS; } - /* Check if we need to write the doorbell */ - if (likely(sq->doorbell.d64 != 0)) { - mlx5e_tx_notify_hw(sq, sq->doorbell.d32); - sq->doorbell.d64 = 0; - } + /* Write the doorbell record, if any. */ + mlx5e_tx_notify_hw(sq, false); /* * Check if we need to start the event timer which flushes the