From nobody Thu Feb 17 12:13:51 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3A1F619C29E6; Thu, 17 Feb 2022 12:13:52 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jztz409W0z4tZQ; Thu, 17 Feb 2022 12:13:51 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1645100032; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=MpSh7O0wDqKCbIap9qY0tbXkiCjDK5JyeyiVEusJjV8=; b=WH04DbnM9U8Z0I5FuoS7TdwH0LiMP/WzS64ct/AOy/9b+wRwf8wKNZ34YqShyyuFxxZ5n0 Ir5UGhDa1M15PGgwBk5lXzDYaD10RPsYfq/iHM0ZIABX2kng/apXSXj8SVHFS3E/HtbWR+ JlLmBz4zBZrSwYd6HwubMMZE7GUvrFyzJIntu9gqv6SLvTxB2vpiTzjptcHJ6b1b+1Wf/b 6VlqQpzSHZ5FVkdTfPIrs3lWGXC5zuuuaexJldQqUKUKDEPf+8l3BQy3wjPSJs1lAr8HO8 W7B5ddb11HzLnQ+i6yYlfkS3MtRxO6NVvezff3BdDCm7vxZjpfzJLDnILnrqQA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id B90751A17; Thu, 17 Feb 2022 12:13:51 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 21HCDpRL027603; Thu, 17 Feb 2022 12:13:51 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 21HCDpsU027602; Thu, 17 Feb 2022 12:13:51 GMT (envelope-from git) Date: Thu, 17 Feb 2022 12:13:51 GMT Message-Id: <202202171213.21HCDpsU027602@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Hans Petter Selasky Subject: git: bc531a1faa99 - main - mlx5en: Improve CQE error debugging. List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: hselasky X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: bc531a1faa99b94b7b7761f1640304dd815eec5d Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1645100032; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=MpSh7O0wDqKCbIap9qY0tbXkiCjDK5JyeyiVEusJjV8=; b=jFffZhNNaN50fYfYm8eUzPsLoyw/tKt015kl6WyTLZDiUorQmnfqGYQK9iOqYLWDwMG3Im ovwBiJhx+MuEoM2lqvTvuVx3OQt6uzT2BDvHn9w/dILC22Rk2zFwOgsTLjxCKIWDj4A8uM IuHcUW89+EwJp7fxd1hr1m9in09BvqWEXcUCh2hw+IrRLLfdsb5wGjHcprqYxc74RryHDX XOMOZlQFhVssTYAaObtqoFGeK2rEfUsQtTb2ux1B7EiG7Q2/oZBgiPEYceDnCSIfzwwutG I4TZMTLzE8Ds1cjlIMYwE3nEHTgadA/+D1DTq6S10XXh4TMD9TQrv70auikz+Q== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1645100032; a=rsa-sha256; cv=none; b=M0kskeBrYw7WoUay4RuVlxEtpRRRyGvGHOAYFSloF0v2SSvee6P/R8ChQkK0irttGzP+BN VpeP8SAtiUweEgiXD5ENMk0/o3OAjB3G8Qvuvz02CFp+QYUCUdADeDhkBVoOzY6nkcNlf9 wNvV9P+Pk1FSqx41Q4nFvJvw5zytDLKo6Od/adrXtPUx8Y8WAJ3icldSa3loXdTy9D5cmN Rx5izLCdnsAnrfBWT3MbSkkQZFbZU+P1rujzubQVviBVrto8XlhkxicIxxeJDh/XW55Qrm vGwY2H773KLReke3ztYmSakS1BikeSHEAvItCypfR3Pw/vbzyYaLcT8MImUihw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by hselasky: URL: https://cgit.FreeBSD.org/src/commit/?id=bc531a1faa99b94b7b7761f1640304dd815eec5d commit bc531a1faa99b94b7b7761f1640304dd815eec5d Author: Hans Petter Selasky AuthorDate: 2022-02-17 11:50:22 +0000 Commit: Hans Petter Selasky CommitDate: 2022-02-17 12:13:09 +0000 mlx5en: Improve CQE error debugging. MFC after: 1 week Sponsored by: NVIDIA Networking --- sys/dev/mlx5/mlx5_en/en.h | 2 ++ sys/dev/mlx5/mlx5_en/mlx5_en_rx.c | 1 + sys/dev/mlx5/mlx5_en/mlx5_en_tx.c | 4 +++- sys/dev/mlx5/mlx5_en/mlx5_en_txrx.c | 25 +++++++++++++++++++++++++ 4 files changed, 31 insertions(+), 1 deletion(-) diff --git a/sys/dev/mlx5/mlx5_en/en.h b/sys/dev/mlx5/mlx5_en/en.h index fa355c68831e..36a55ff5c4d0 100644 --- a/sys/dev/mlx5/mlx5_en/en.h +++ b/sys/dev/mlx5/mlx5_en/en.h @@ -1205,6 +1205,8 @@ int mlx5e_open_locked(struct ifnet *); int mlx5e_close_locked(struct ifnet *); void mlx5e_cq_error_event(struct mlx5_core_cq *mcq, int event); +void mlx5e_dump_err_cqe(struct mlx5e_cq *, u32, const struct mlx5_err_cqe *); + mlx5e_cq_comp_t mlx5e_rx_cq_comp; mlx5e_cq_comp_t mlx5e_tx_cq_comp; struct mlx5_cqe64 *mlx5e_get_cqe(struct mlx5e_cq *cq); diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c b/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c index 0e3a3b3917f4..3c8813190f76 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c @@ -495,6 +495,7 @@ mlx5e_poll_rx_cq(struct mlx5e_rq *rq, int budget) BUS_DMASYNC_POSTREAD); if (unlikely((cqe->op_own >> 4) != MLX5_CQE_RESP_SEND)) { + mlx5e_dump_err_cqe(&rq->cq, rq->rqn, (const void *)cqe); rq->stats.wqe_err++; goto wq_ll_pop; } diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c b/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c index 9e0837a76393..78458ab69f13 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_tx.c @@ -1045,8 +1045,10 @@ mlx5e_poll_tx_cq(struct mlx5e_sq *sq, int budget) mlx5_cqwq_pop(&sq->cq.wq); /* check if the completion event indicates an error */ - if (unlikely(get_cqe_opcode(cqe) != MLX5_CQE_REQ)) + if (unlikely(get_cqe_opcode(cqe) != MLX5_CQE_REQ)) { + mlx5e_dump_err_cqe(&sq->cq, sq->sqn, (const void *)cqe); sq->stats.cqe_err++; + } /* setup local variables */ sqcc_this = be16toh(cqe->wqe_counter); diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_txrx.c b/sys/dev/mlx5/mlx5_en/mlx5_en_txrx.c index 9f5e17ad864e..aff247f5aea2 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_txrx.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_txrx.c @@ -28,6 +28,8 @@ #include "opt_rss.h" #include "opt_ratelimit.h" +#include + #include struct mlx5_cqe64 * @@ -54,3 +56,26 @@ mlx5e_cq_error_event(struct mlx5_core_cq *mcq, int event) mlx5_en_err(cq->priv->ifp, "cqn=0x%.6x event=0x%.2x\n", mcq->cqn, event); } + +void +mlx5e_dump_err_cqe(struct mlx5e_cq *cq, u32 qn, const struct mlx5_err_cqe *err_cqe) +{ + u32 ci; + + /* Don't print flushed in error syndromes. */ + if (err_cqe->vendor_err_synd == 0xf9 && err_cqe->syndrome == 0x05) + return; + /* Don't print when the queue is set to error state by software. */ + if (err_cqe->vendor_err_synd == 0xf5 && err_cqe->syndrome == 0x05) + return; + + ci = (cq->wq.cc - 1) & cq->wq.sz_m1; + + mlx5_en_err(cq->priv->ifp, + "Error CQE on CQN 0x%x, CI 0x%x, QN 0x%x, OPCODE 0x%x, SYNDROME 0x%x, VENDOR SYNDROME 0x%x\n", + cq->mcq.cqn, ci, qn, err_cqe->op_own >> 4, + err_cqe->syndrome, err_cqe->vendor_err_synd); + + print_hex_dump(NULL, NULL, DUMP_PREFIX_OFFSET, + 16, 1, err_cqe, sizeof(*err_cqe), false); +}