From nobody Wed Jul 06 15:06:34 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3314B1CFC5D0; Wed, 6 Jul 2022 15:06:35 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LdNDB69Wfz4hCJ; Wed, 6 Jul 2022 15:06:34 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1657119995; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=LBUQWhua5LjQDh8E4Vj+ihCnFlRl1zihxZRqZihlbRI=; b=OGVg2S2CyUVVxfeoYKls/75+XFxJ1TwP7zLgfo+sODh3TGbbFjjKF4Y5pOucHFCf7oSW8E lfBP6hRGv2oyq7MZtiejewPsxIofk5wh47FQSvrcbkLWRmsjOzfc/G39Wb4MNw4Vrlg7xn sJekllvdE1jNMiqokZzElqBbLSbrouo2qhRbmMltW2hJ2flkSP6/7ctKVAGDsDQwa+hFJr Uu3ZMgIOd0zVUL6aJL2V0y1XgozDU2pZOtBKAhjQjntCL9w7x5RtOwaUf5DSg51IlVTdkc AhLEcTA+XYxNxwhMDDVEM9jToqqhO9VTq1g2BinOTNJv3kQLMpzb87c9HaV19g== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8E4D41CAAF; Wed, 6 Jul 2022 15:06:34 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 266F6YdB020835; Wed, 6 Jul 2022 15:06:34 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 266F6Y31020834; Wed, 6 Jul 2022 15:06:34 GMT (envelope-from git) Date: Wed, 6 Jul 2022 15:06:34 GMT Message-Id: <202207061506.266F6Y31020834@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Marcin Wojtas Subject: git: 3324e304c14d - main - ena: Fix LLQ descriptor reconfiguration List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mw X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 3324e304c14d121dc04338a38c7f3277cbfba58b Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1657119995; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=LBUQWhua5LjQDh8E4Vj+ihCnFlRl1zihxZRqZihlbRI=; b=D2Rmy5ddHV48ZQZfuznQZfLOGeRVrh+pgIS1J78Ef0kwlCh4oTY/e0Xge/V9+Kdk898/bE dQ5X864REJIvMqQOUOfYju59a4VBG9tBtp5wV+aK6Qjg0qisomnKJJ1tF6hpmsrlrWRvTk 55ClOko8Ml5PoxUZlgEQhGep6RDiFNgS1XJqClwhMPkfkM8c+NRHS0WGhKX2myGWVbvL4n yvO0HN7YGwAhvENAS7Rvc2kgmodkV7FS8Ss6HQzrMrSHIXkYH/y1vUzAcKwVRZFkY9LmSH I/rqcPVK28Uwe5XTZRycWI37k5q5H95MFKpmj1a4WE7VDh75ugBcCQHJr9whSw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1657119995; a=rsa-sha256; cv=none; b=vvRWVmdZ5eujh/PHLPvsgdj1JQ79m8eV+X+d1346+SVclGlfrFtXbF4qjIft689vtArcH4 VNzX9rZmMeofCW/syQKQxgNF5tfmWKE96hV9wWxjbjcBn5ca43RJep+Hk6ZVzAU1GmdUrO FGEiQFwLZCFxbOaTqZBMKjGbcQaGa29S9i67EgxRqHZ8w/XY9/6doOCg9qYX8ImIkaHCM8 x/IsEQ/aF6my2oIT4MSellLeaMZTjDdp9dU/TxpcYSv2NYQY7Uop8nQ36mJC5h35RBeW9R 0+1t8U8B5L7Rv0smZFCCfTWb2JUFxShbTeSgiG5tFnUaMmhfYiE1o1Wdb0xMuw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by mw: URL: https://cgit.FreeBSD.org/src/commit/?id=3324e304c14d121dc04338a38c7f3277cbfba58b commit 3324e304c14d121dc04338a38c7f3277cbfba58b Author: Michal Krawczyk AuthorDate: 2022-07-05 10:59:25 +0000 Commit: Marcin Wojtas CommitDate: 2022-07-06 15:06:21 +0000 ena: Fix LLQ descriptor reconfiguration After the device reset, the LLQ configuration descriptor wasn't passed to the hardware. On a 6-generation AWS instances (like C6gn), it is required to pass the LLQ descriptor after the device reset, otherwise the hardware will be missing the LLQ configuration resulting in performance degradation. This patch reconfigures the LLQ each time the ena_device_init() is called. This means that the LLQ descriptor will be passed during the initial configuration and after a reset. The ena_map_llq_mem_bar() function call was moved before the ena_device_init() call, to make sure that the mem bar is available. Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. --- sys/dev/ena/ena.c | 35 ++++++++++++++++------------------- 1 file changed, 16 insertions(+), 19 deletions(-) diff --git a/sys/dev/ena/ena.c b/sys/dev/ena/ena.c index d032da6e2206..c091091fed20 100644 --- a/sys/dev/ena/ena.c +++ b/sys/dev/ena/ena.c @@ -2579,8 +2579,7 @@ ena_map_llq_mem_bar(device_t pdev, struct ena_com_dev *ena_dev) RF_ACTIVE); if (unlikely(adapter->memory == NULL)) { ena_log(pdev, WARN, - "unable to allocate LLQ bar resource. Fallback to host mode policy.\n"); - ena_dev->tx_mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_HOST; + "Unable to allocate LLQ bar resource. LLQ mode won't be used.\n"); return (0); } @@ -2765,6 +2764,7 @@ static int ena_device_init(struct ena_adapter *adapter, device_t pdev, struct ena_com_dev_get_features_ctx *get_feat_ctx, int *wd_active) { + struct ena_llq_configurations llq_config; struct ena_com_dev *ena_dev = adapter->ena_dev; bool readless_supported; uint32_t aenq_groups; @@ -2844,6 +2844,15 @@ ena_device_init(struct ena_adapter *adapter, device_t pdev, *wd_active = !!(aenq_groups & BIT(ENA_ADMIN_KEEP_ALIVE)); + set_default_llq_configurations(&llq_config, &get_feat_ctx->llq); + + rc = ena_set_queues_placement_policy(pdev, ena_dev, &get_feat_ctx->llq, + &llq_config); + if (unlikely(rc != 0)) { + ena_log(pdev, ERR, "Failed to set placement policy\n"); + goto err_admin_init; + } + return (0); err_admin_init: @@ -3487,7 +3496,6 @@ static int ena_attach(device_t pdev) { struct ena_com_dev_get_features_ctx get_feat_ctx; - struct ena_llq_configurations llq_config; struct ena_calc_queue_size_ctx calc_queue_ctx = { 0 }; static int version_printed; struct ena_adapter *adapter; @@ -3560,7 +3568,11 @@ ena_attach(device_t pdev) goto err_bus_free; } - ena_dev->tx_mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_HOST; + rc = ena_map_llq_mem_bar(pdev, ena_dev); + if (unlikely(rc != 0)) { + ena_log(pdev, ERR, "Failed to map ENA mem bar"); + goto err_bus_free; + } /* Initially clear all the flags */ ENA_FLAG_ZERO(adapter); @@ -3573,21 +3585,6 @@ ena_attach(device_t pdev) goto err_bus_free; } - set_default_llq_configurations(&llq_config, &get_feat_ctx.llq); - - rc = ena_map_llq_mem_bar(pdev, ena_dev); - if (unlikely(rc != 0)) { - ena_log(pdev, ERR, "failed to map ENA mem bar"); - goto err_com_free; - } - - rc = ena_set_queues_placement_policy(pdev, ena_dev, &get_feat_ctx.llq, - &llq_config); - if (unlikely(rc != 0)) { - ena_log(pdev, ERR, "failed to set placement policy\n"); - goto err_com_free; - } - if (ena_dev->tx_mem_queue_type == ENA_ADMIN_PLACEMENT_POLICY_DEV) adapter->disable_meta_caching = !!( get_feat_ctx.llq.accel_mode.u.get.supported_flags &