From nobody Tue Feb 01 15:24:06 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D36701914EAF; Tue, 1 Feb 2022 15:24:06 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jp7xy53chz3Fph; Tue, 1 Feb 2022 15:24:06 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1643729046; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=BnaX0lAoLQ4MQWFr0Jbz5XzhMq1YEzVuI19jSRNNHc4=; b=WFoAQUaN9BJNT02Nv9FXH6rPisW88kHrR10Z3OJtDK7nJtOCNB8LY0X3e2tukv7lzz3lFT yoq/Xd7ghX7Vtm10qFLoacELm6ER3DA8Sfj8Gneni4chrj2i5OZAjPHlLYYoXzyC1TAo88 6e6VgrGSUvPBktmGx/iLtGylv4XH1BD/T4J9qk9MXAiCI1zxwfcD1QHolEsdT0JcQfI1NL gU9ay8suK/ElPVJvBaI8hgVX27RfVGIspc6OOk7EElIzPl72yKE7SehZV1vP8a71PWyt2/ vFhmrdaJdd8+mapoP13i5gpTP7UpsVVeuXqbquKC50NGbMzbm/XOzLdYvV2waw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 87E0913173; Tue, 1 Feb 2022 15:24:06 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 211FO6JB050562; Tue, 1 Feb 2022 15:24:06 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 211FO6HM050561; Tue, 1 Feb 2022 15:24:06 GMT (envelope-from git) Date: Tue, 1 Feb 2022 15:24:06 GMT Message-Id: <202202011524.211FO6HM050561@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Hans Petter Selasky Subject: git: 2d5e5a0d75b0 - main - mlx5en: Patch to inhibit transmit doorbell writes during packet reception. List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: hselasky X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 2d5e5a0d75b0612f8cea0f7f59f7d2e59b0c7a60 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1643729046; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=BnaX0lAoLQ4MQWFr0Jbz5XzhMq1YEzVuI19jSRNNHc4=; b=qeQFhSHC44MjuyBAdIHowVfUcfPdh3ol1C5t55kH15B+e4yrN9PJBjElBqqx2nXWRJdqex I3oBqLnyAaTrFNpfmN7WfAd3DqMjgNMzuvZyojBO+wwC/N/FeuR+zlOQ4K/bIJSlA5mcs0 l/4GlltncjJh6bdvtE0cne7s4v0VLBgEgSQE7wMA2gRXsw/DXuk1842BuyNDSKteMTjRKv Jge+ewD5eNfzHBnQFu6bCNHnHfKLlkc8c1w0DpfTOo2zpwkjsf4WZM/qxHQP9xh+L+1h+y OI6LGbgiKLcs28bkNpdlLWUz0322SwOqIkSLEFOzuE7u73RRII/iVcAJ+2G7Sg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1643729046; a=rsa-sha256; cv=none; b=P4hgSIIUt4vEOZgV/3gzhbSLCHMVP/8F9g3rLNaLzTDdgJlZQGkFrkQNDcndA92g5BA5E2 wxfG0ekRx1U/0wNy8XhAOy7Au4jjlmIc1elZm1YVAz0QSi5mq397wpDpWmZCXMhKPGLDiq tR4QLW8Qp3oqAAtB1cN5xftYsqeWd8w7iqy3AjpKehKmSS9on0Va6BXm/mSl+xxqPXCTct YSkxAwN/9YaSLwnUYbcQP7/MfVkDB1yW2a0nWrRGzSsanD7kv24lNfH0yQdfS7VH6OrPDP EJ+XVEep8XHdpxS9YI3nNVe+SpViKp9btvIEqfn6cTP/5W6r30SvBlhmOglyOQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by hselasky: URL: https://cgit.FreeBSD.org/src/commit/?id=2d5e5a0d75b0612f8cea0f7f59f7d2e59b0c7a60 commit 2d5e5a0d75b0612f8cea0f7f59f7d2e59b0c7a60 Author: Hans Petter Selasky AuthorDate: 2022-02-01 15:20:04 +0000 Commit: Hans Petter Selasky CommitDate: 2022-02-01 15:21:13 +0000 mlx5en: Patch to inhibit transmit doorbell writes during packet reception. During packet reception the network stack frequently transmit data in response to TCP window updates. To reduce the number of transmit doorbells needed, inhibit all transmit doorbells designated for the same channel until after the reception of packets for the given channel is completed. While at it slightly refactor the mlx5e_tx_notify_hw() function: 1) The doorbell information is always stored into sq->doorbell.d64 . No need to pass a separate pointer to this variable. 2) Move checks for skipping doorbell writes inside this function. MFC after: 1 week Sponsored by: NVIDIA Networking --- sys/dev/mlx5/mlx5_en/en.h | 12 ++++++++++-- sys/dev/mlx5/mlx5_en/mlx5_en_main.c | 15 ++++----------- sys/dev/mlx5/mlx5_en/mlx5_en_rx.c | 16 +++++++++++++++- sys/dev/mlx5/mlx5_en/mlx5_en_tx.c | 9 +++------ 4 files changed, 32 insertions(+), 20 deletions(-) diff --git a/sys/dev/mlx5/mlx5_en/en.h b/sys/dev/mlx5/mlx5_en/en.h index af6cd8371506..4eb771d5c5c6 100644 --- a/sys/dev/mlx5/mlx5_en/en.h +++ b/sys/dev/mlx5/mlx5_en/en.h @@ -797,6 +797,7 @@ struct mlx5e_sq { struct mtx comp_lock; struct mlx5e_sq_stats stats; struct callout cev_callout; + int db_inhibit; /* data path */ #define mlx5e_sq_zero_start dma_tag @@ -1154,8 +1155,13 @@ int mlx5e_add_all_vxlan_rules(struct mlx5e_priv *priv); void mlx5e_del_all_vxlan_rules(struct mlx5e_priv *priv); static inline void -mlx5e_tx_notify_hw(struct mlx5e_sq *sq, u32 *wqe) +mlx5e_tx_notify_hw(struct mlx5e_sq *sq, bool force) { + if (unlikely((force == false && sq->db_inhibit != 0) || sq->doorbell.d64 == 0)) { + /* skip writing the doorbell record */ + return; + } + /* ensure wqe is visible to device before updating doorbell record */ wmb(); @@ -1167,8 +1173,10 @@ mlx5e_tx_notify_hw(struct mlx5e_sq *sq, u32 *wqe) */ wmb(); - mlx5_write64(wqe, sq->uar_map, + mlx5_write64(sq->doorbell.d32, sq->uar_map, MLX5_GET_DOORBELL_LOCK(&sq->priv->doorbell_lock)); + + sq->doorbell.d64 = 0; } static inline void diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c index e8d9d26efc6f..6eb266f37761 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c @@ -1807,11 +1807,7 @@ mlx5e_sq_send_nops_locked(struct mlx5e_sq *sq, int can_sleep) atomic_thread_fence_rel(); } done: - /* Check if we need to write the doorbell */ - if (likely(sq->doorbell.d64 != 0)) { - mlx5e_tx_notify_hw(sq, sq->doorbell.d32); - sq->doorbell.d64 = 0; - } + mlx5e_tx_notify_hw(sq, false); } void @@ -3867,8 +3863,7 @@ mlx5e_reset_sq_doorbell_record(struct mlx5e_sq *sq) sq->doorbell.d32[0] = cpu_to_be32(MLX5_OPCODE_NOP); sq->doorbell.d32[1] = cpu_to_be32(sq->sqn << 8); - mlx5e_tx_notify_hw(sq, sq->doorbell.d32); - sq->doorbell.d64 = 0; + mlx5e_tx_notify_hw(sq, true); } void @@ -4785,10 +4780,8 @@ mlx5_en_debugnet_transmit(struct ifnet *dev, struct mbuf *m) err = 0; } - if (likely(sq->doorbell.d64 != 0)) { - mlx5e_tx_notify_hw(sq, sq->doorbell.d32); - sq->doorbell.d64 = 0; - } + mlx5e_tx_notify_hw(sq, true); + return (err); } diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c b/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c index e943a65709cf..ec6b027b324e 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2015 Mellanox Technologies. All rights reserved. + * Copyright (c) 2015-2021 Mellanox Technologies. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions @@ -566,6 +566,7 @@ wq_ll_pop: void mlx5e_rx_cq_comp(struct mlx5_core_cq *mcq, struct mlx5_eqe *eqe __unused) { + struct mlx5e_channel *c = container_of(mcq, struct mlx5e_channel, rq.cq.mcq); struct mlx5e_rq *rq = container_of(mcq, struct mlx5e_rq, cq.mcq); int i = 0; @@ -584,6 +585,11 @@ mlx5e_rx_cq_comp(struct mlx5_core_cq *mcq, struct mlx5_eqe *eqe __unused) rq->ifp->if_input(rq->ifp, mb); } #endif + for (int j = 0; j != MLX5E_MAX_TX_NUM_TC; j++) { + mtx_lock(&c->sq[j].lock); + c->sq[j].db_inhibit++; + mtx_unlock(&c->sq[j].lock); + } mtx_lock(&rq->mtx); @@ -607,4 +613,12 @@ mlx5e_rx_cq_comp(struct mlx5_core_cq *mcq, struct mlx5_eqe *eqe __unused) mlx5e_cq_arm(&rq->cq, MLX5_GET_DOORBELL_LOCK(&rq->channel->priv->doorbell_lock)); tcp_lro_flush_all(&rq->lro); mtx_unlock(&rq->mtx); + + for (int j = 0; j != MLX5E_MAX_TX_NUM_TC; j++) { + mtx_lock(&c->sq[j].lock); + c->sq[j].db_inhibit--; + /* Update the doorbell record, if any. */ + mlx5e_tx_notify_hw(c->sq + j, true); + mtx_unlock(&c->sq[j].lock); + } } diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c b/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c index 7bb12d7b5b8a..1bf3a857d1e8 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c @@ -1,5 +1,5 @@ /*- - * Copyright (c) 2015-2019 Mellanox Technologies. All rights reserved. + * Copyright (c) 2015-2021 Mellanox Technologies. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions @@ -1122,11 +1122,8 @@ mlx5e_xmit_locked(struct ifnet *ifp, struct mlx5e_sq *sq, struct mbuf *mb) err = ENOBUFS; } - /* Check if we need to write the doorbell */ - if (likely(sq->doorbell.d64 != 0)) { - mlx5e_tx_notify_hw(sq, sq->doorbell.d32); - sq->doorbell.d64 = 0; - } + /* Write the doorbell record, if any. */ + mlx5e_tx_notify_hw(sq, false); /* * Check if we need to start the event timer which flushes the