From nobody Thu Feb 24 10:04:21 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 37B1519E017F; Thu, 24 Feb 2022 10:04:23 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K47mQ4Bmlz3QGB; Thu, 24 Feb 2022 10:04:22 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1645697062; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=lRyOiFRn3bLnBznVXRdmiYwUdvtAAfRrFfGDSJg3ugY=; b=K+LgFoVRROLRyqVLmpCctXAUtf2A9ApK4uJv5D7m/aEn5RE53gMrhwsVzcgkdSbEJARLiG vE6UpWSoRALi6AOk6awvzuxP7rX9MXo60ab6ktkHrtjRhdM2a39O8WADVos6vh26LnWJ+Y pCVoN4Sk6ODB84kZkM3mnIWjcBwzLezVvlymqSYVE9MNE0VIRwhPFKdaPKFt76wspmjWym 4I34iUfz2SKTqcwUYHdvPZ5IRhd/Bm49neogN1NKCr0fNKW5O4fdlH/JhRGzHwSlsGgxHo sgBxRtNKXSHXXfTanrBW8jHuG7T6CJhpXIi+KBgGY5ycORhzJnC8u0I80ZzgEw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0970F1A7E; Thu, 24 Feb 2022 10:04:22 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 21OA4LBD003595; Thu, 24 Feb 2022 10:04:21 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 21OA4L3X003594; Thu, 24 Feb 2022 10:04:21 GMT (envelope-from git) Date: Thu, 24 Feb 2022 10:04:21 GMT Message-Id: <202202241004.21OA4L3X003594@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Hans Petter Selasky Subject: git: 4b5dd427cb63 - stable/13 - mlx5en: Improve CQE error debugging. List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: hselasky X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 4b5dd427cb63a2f3965d64afe4cc07f35189639a Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1645697062; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=lRyOiFRn3bLnBznVXRdmiYwUdvtAAfRrFfGDSJg3ugY=; b=rTmNXVcm3ysahPldYMtB3iM+OrMzLsu1nmQAe/nLNfs5C8usbw+dF6cepM5CjNToNWu81s OHpRyFhurHeXILqerXRxAlnt/15ALGBE1l/jFSXYlJHCMPaY4h0jeMpY7iRdzXNwUDys/Y XCSwgtjuZmsH9xP4LF8u1nmcJ3J5OshDB2TDNswM0NcMyN6eImPhvC3VNakU0ahOAoA01R LBEj0GthbaIIEFEEipJwSz3SEve8vWjls2zi7uOqjEuij10InDcrjEqs/BKYfFXX9ASLMT C9RkQrjhAQEkD2KgLU4qrplcoMT3sLIMDlxe3PJfY3URX0rrD+iErSbyzvBpkg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1645697062; a=rsa-sha256; cv=none; b=RnA1UxgY3JV6ToyBEPSaRIskHEdzF4PcBCKYRZ7pxVY/x4qYjnUZLESDiQB6zaHlKkWgEy V+5/BGiX9VSFuekti7iuJM0OBePKizdYhLLueDkcTtDpb51Ny/VYuGLdacaro1brxOBD+q wTc+rZmW2gbsEebI/k201ai53csov+uKFfu15OzPfWeIrsECHDbNmrldpu8C8wiQ7jN7Z+ T/E1pWollJyZ9ppBKiRVrOIYEb9Vcytzq4akRlDoEshe5WXdiHMCXUnbcjL97cCA5N2UNx 2iMysPq0lihEDqYGugIz71B6OPtEqb5TRO5KcNWj3UeTXT5M5a8vYbSMKH/PRw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by hselasky: URL: https://cgit.FreeBSD.org/src/commit/?id=4b5dd427cb63a2f3965d64afe4cc07f35189639a commit 4b5dd427cb63a2f3965d64afe4cc07f35189639a Author: Hans Petter Selasky AuthorDate: 2022-02-24 09:58:54 +0000 Commit: Hans Petter Selasky CommitDate: 2022-02-24 09:58:54 +0000 mlx5en: Improve CQE error debugging. Sponsored by: NVIDIA Networking (cherry picked from commit bc531a1faa99b94b7b7761f1640304dd815eec5d) --- sys/dev/mlx5/mlx5_en/en.h | 2 ++ sys/dev/mlx5/mlx5_en/mlx5_en_rx.c | 1 + sys/dev/mlx5/mlx5_en/mlx5_en_tx.c | 4 +++- sys/dev/mlx5/mlx5_en/mlx5_en_txrx.c | 25 +++++++++++++++++++++++++ 4 files changed, 31 insertions(+), 1 deletion(-) diff --git a/sys/dev/mlx5/mlx5_en/en.h b/sys/dev/mlx5/mlx5_en/en.h index 405fe6390cfb..28928aa18fa1 100644 --- a/sys/dev/mlx5/mlx5_en/en.h +++ b/sys/dev/mlx5/mlx5_en/en.h @@ -1198,6 +1198,8 @@ int mlx5e_open_locked(struct ifnet *); int mlx5e_close_locked(struct ifnet *); void mlx5e_cq_error_event(struct mlx5_core_cq *mcq, int event); +void mlx5e_dump_err_cqe(struct mlx5e_cq *, u32, const struct mlx5_err_cqe *); + mlx5e_cq_comp_t mlx5e_rx_cq_comp; mlx5e_cq_comp_t mlx5e_tx_cq_comp; struct mlx5_cqe64 *mlx5e_get_cqe(struct mlx5e_cq *cq); diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c b/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c index 9e58a9bd7904..9de80265f078 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c @@ -482,6 +482,7 @@ mlx5e_poll_rx_cq(struct mlx5e_rq *rq, int budget) BUS_DMASYNC_POSTREAD); if (unlikely((cqe->op_own >> 4) != MLX5_CQE_RESP_SEND)) { + mlx5e_dump_err_cqe(&rq->cq, rq->rqn, (const void *)cqe); rq->stats.wqe_err++; goto wq_ll_pop; } diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c b/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c index 24899ef0b881..4cc25e1d5b51 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c @@ -1042,8 +1042,10 @@ mlx5e_poll_tx_cq(struct mlx5e_sq *sq, int budget) mlx5_cqwq_pop(&sq->cq.wq); /* check if the completion event indicates an error */ - if (unlikely(get_cqe_opcode(cqe) != MLX5_CQE_REQ)) + if (unlikely(get_cqe_opcode(cqe) != MLX5_CQE_REQ)) { + mlx5e_dump_err_cqe(&sq->cq, sq->sqn, (const void *)cqe); sq->stats.cqe_err++; + } /* setup local variables */ sqcc_this = be16toh(cqe->wqe_counter); diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_txrx.c b/sys/dev/mlx5/mlx5_en/mlx5_en_txrx.c index 9f5e17ad864e..aff247f5aea2 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_txrx.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_txrx.c @@ -28,6 +28,8 @@ #include "opt_rss.h" #include "opt_ratelimit.h" +#include + #include struct mlx5_cqe64 * @@ -54,3 +56,26 @@ mlx5e_cq_error_event(struct mlx5_core_cq *mcq, int event) mlx5_en_err(cq->priv->ifp, "cqn=0x%.6x event=0x%.2x\n", mcq->cqn, event); } + +void +mlx5e_dump_err_cqe(struct mlx5e_cq *cq, u32 qn, const struct mlx5_err_cqe *err_cqe) +{ + u32 ci; + + /* Don't print flushed in error syndromes. */ + if (err_cqe->vendor_err_synd == 0xf9 && err_cqe->syndrome == 0x05) + return; + /* Don't print when the queue is set to error state by software. */ + if (err_cqe->vendor_err_synd == 0xf5 && err_cqe->syndrome == 0x05) + return; + + ci = (cq->wq.cc - 1) & cq->wq.sz_m1; + + mlx5_en_err(cq->priv->ifp, + "Error CQE on CQN 0x%x, CI 0x%x, QN 0x%x, OPCODE 0x%x, SYNDROME 0x%x, VENDOR SYNDROME 0x%x\n", + cq->mcq.cqn, ci, qn, err_cqe->op_own >> 4, + err_cqe->syndrome, err_cqe->vendor_err_synd); + + print_hex_dump(NULL, NULL, DUMP_PREFIX_OFFSET, + 16, 1, err_cqe, sizeof(*err_cqe), false); +}