From nobody Mon Aug 28 09:17:38 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZ4hg114nz4rLR9; Mon, 28 Aug 2023 09:17:39 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZ4hg0V5Jz3NsT; Mon, 28 Aug 2023 09:17:39 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693214259; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=QWfE2tKDjVyDkAyoNl9hvIWyRzQdsrm9Tb0KT5gyGEw=; b=lmPLlekbxdBSvJ3Q7r0i6pFzQ38iazWnl+NA9RmlhZO7Qg6ToTEXU+KwWGHpyu5t6C6F59 O4lDJcgllabP20GdlUmn/uW3YhSFg3c400ArULyrrinVZZGJB5WSh34jyO11aX8XH1lHiU kXkba/gvi4i/uOrDNMZlk48fSRlbNd7WCUtUNVMJPq3sPOLxfHqfVP2pgtr6Dmp9xt89EC bh3DtxFsVGkzETmR6ZYQQXE+fvUQ++Egca8/gpn48fm20Fxr3H/K7a54QNEUj7lvVjGp+K nL+0nFptUoSa/+CW1iKDNXLtIuE0iGHf+Qah80Ui+Cs1WI2ZUcs1TJGhLEYgTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693214259; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=QWfE2tKDjVyDkAyoNl9hvIWyRzQdsrm9Tb0KT5gyGEw=; b=sF+APpM8Yqmw0wWzYiABBPj9dUuADqnae1aTi/nlaRv0ukCVA366OHiq+vsCUqxcgBREoC wvPoum5DfQLMziW0bvXtEuYPpJIJeCdSLu26j7mQvY0qIQuajagS3Zxbczuc77B+OytLY+ 6kj78G+kG9xDGTXoynwSafbiPk4WjnILjmTLMy8Oxj3dFH5umoImk7TObv5JVh6mo7UuXG 1g990u32uNi5F6U2oy+8RUurAA5lDce/t4iEjPr3u65i0i5BtkQfNatmF9xY5OkM3UAIZl 9JaY0HJWZxD2uQWvAEFfhHdKjj4z8m1j2PTveYr12CWncBbCPKpyE2IDcvoirw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693214259; a=rsa-sha256; cv=none; b=TPYP9m6bVOp9Qy5vftKTYReo3b7mtqkYeNy7aIhqububg87+gqirEm9jm4fcJ9Dek+HMLE VS+fIRYVegRgUhGui8xHP7ENdkEgR8ZITFlHdihRt2EtF2YaQIiZhC39+vqG9yOCx8d5HU MMKgB8NATOm9IGChs6wIksWlHPFbWlVC904AqjkfaPYCXs03E5zZTNbgnUQydlVqCdqHMi dpetYKfOG/ioBtwHGLPg7zE7QLnoHCsHdvSjiD+3J8hv72645jhT4I2ajsZ2jWJW0Dr3Ux /Eus4TQh5nOk56fW9SA5hdtKUO9Y5gpklCo8L21mfXNHurFusEJlH9WMdlFRZg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RZ4hf6dg7z13K2; Mon, 28 Aug 2023 09:17:38 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 37S9Hcjl049289; Mon, 28 Aug 2023 09:17:38 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 37S9HcuQ049286; Mon, 28 Aug 2023 09:17:38 GMT (envelope-from git) Date: Mon, 28 Aug 2023 09:17:38 GMT Message-Id: <202308280917.37S9HcuQ049286@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Wei Hu Subject: git: e4e11c1d07f5 - main - mana: batch ringing RX queue doorbell on receiving packets List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: whu X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: e4e11c1d07f5d58ff8cf4e07ac8f61eecbbb5417 Auto-Submitted: auto-generated The branch main has been updated by whu: URL: https://cgit.FreeBSD.org/src/commit/?id=e4e11c1d07f5d58ff8cf4e07ac8f61eecbbb5417 commit e4e11c1d07f5d58ff8cf4e07ac8f61eecbbb5417 Author: Wei Hu AuthorDate: 2023-08-28 09:15:16 +0000 Commit: Wei Hu CommitDate: 2023-08-28 09:15:16 +0000 mana: batch ringing RX queue doorbell on receiving packets It's inefficient to ring the doorbell page every time a WQE is posted to the received queue. Excessive MMIO writes result in CPU spending more time waiting on LOCK instructions (atomic operations), resulting in poor scaling performance. Move the code for ringing doorbell page to where after we have posted all WQEs to the receive queue in mana_poll_rx_cq(). In addition, use the correct WQE count for ringing RQ doorbell. The hardware specification specifies that WQE_COUNT should set to 0 for the Receive Queue. Although currently the hardware doesn't enforce the check, in the future releases it may check on this value. Tested by: whu MFC after: 1 week Sponsored by: Microsoft --- sys/dev/mana/gdma_main.c | 2 +- sys/dev/mana/mana_en.c | 9 ++++++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/sys/dev/mana/gdma_main.c b/sys/dev/mana/gdma_main.c index 2de606d54957..a601873876cb 100644 --- a/sys/dev/mana/gdma_main.c +++ b/sys/dev/mana/gdma_main.c @@ -471,7 +471,7 @@ void mana_gd_wq_ring_doorbell(struct gdma_context *gc, struct gdma_queue *queue) { mana_gd_ring_doorbell(gc, queue->gdma_dev->doorbell, queue->type, - queue->id, queue->head * GDMA_WQE_BU_SIZE, 1); + queue->id, queue->head * GDMA_WQE_BU_SIZE, 0); } void diff --git a/sys/dev/mana/mana_en.c b/sys/dev/mana/mana_en.c index 49558cdc97c6..fa49e06e4862 100644 --- a/sys/dev/mana/mana_en.c +++ b/sys/dev/mana/mana_en.c @@ -1524,7 +1524,7 @@ mana_post_pkt_rxq(struct mana_rxq *rxq) recv_buf_oob = &rxq->rx_oobs[curr_index]; - err = mana_gd_post_and_ring(rxq->gdma_rq, &recv_buf_oob->wqe_req, + err = mana_gd_post_work_request(rxq->gdma_rq, &recv_buf_oob->wqe_req, &recv_buf_oob->wqe_inf); if (err) { mana_err(NULL, "WARNING: rxq %u post pkt err %d\n", @@ -1757,6 +1757,13 @@ mana_poll_rx_cq(struct mana_cq *cq) mana_process_rx_cqe(cq->rxq, cq, &comp[i]); } + if (comp_read > 0) { + struct gdma_context *gc = + cq->rxq->gdma_rq->gdma_dev->gdma_context; + + mana_gd_wq_ring_doorbell(gc, cq->rxq->gdma_rq); + } + tcp_lro_flush_all(&cq->rxq->lro); }