From nobody Thu Dec 28 13:57:19 2023 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T19740FKTz55tHC; Thu, 28 Dec 2023 13:57:20 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T19735NRTz4X9p; Thu, 28 Dec 2023 13:57:19 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703771839; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=H2AGAy6wn1MrXzqJQ0LONYURb08rigy+Gn/cVf/afTg=; b=v+v7qNgErp4mRbJcR9TxCzjM9TIfsDPAAy6nihgrvDiV0n7hQet/dNMrp3FO6mJtAsDExQ mti0z81iZ066Cac3etl7TBDV73htfiCOYk/iycyK1pLoL6MOVAqVhvhY5bIUVo64Tu54hy BfGxBZVdyIR31hJjomBM//luKRGGHLqBsH0VIk5HfGtFQxEO8xTlXV1awCtEZ9NlBKIVLr CtYQGnyqjE/DQhviDiy0eNYTKjEL1VjXmMWvllk3GD+ni4Mb1gnhTMVQ9slkzLlefpa4Sa yoxdfNxjU5cGOh1O4P7Ohacm/rVjIgXQwt6RhgaZNMeo/TMGc8dXBqVezUpM6g== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703771839; a=rsa-sha256; cv=none; b=CNxtzxH9v4pEyi0P2IuHs2PrSV6L0oQCkwT50qKi6VWRFdwVoM7WRaw9yJInYIy9DAbAAG Qdl7iS7ISF5UiYjviLcmJQUJ/H9iO5Eo3WB8uG6jTzdkATawOhUpfmbC1QN1BuzvWSYuP/ t81+WX9+bdXVMXvXCEycH0B0zPfhVO6GHk27xSI1FuZxBWYLpBu75nXbIcN87AEIMGPHyx nHAdzwZjY/q3ZU/mJPutrsor0U5p6vTnU4PO965m1kwUjro7luhSW86qRqsZRefDH/7F5c lp/0dVIBOBEnQMGkjCFHxrf4f34TZvKO+UfZH3BvH6p+L+kwL3t5jg/zgliJcA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703771839; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=H2AGAy6wn1MrXzqJQ0LONYURb08rigy+Gn/cVf/afTg=; b=ShsOAhDQo18parwiHGo+OiMpwCKih9RmmjS5WiawOMvdSE2v7y1IeRtaZtaWCrZdZibeM5 iKLHqlWdBO09BBfz5sEFqUm9/kgPSyujO+bz6w/aUvB2G5r+xBxtD6b4jAIbOKsKWD7lJ8 ooZi5Zez9Bkjr77XF3ZI7czJzWjmX+7Vvr11DqaRgUpCfcQ1uz5oG6ysFl+ksN2NWjkm5E Eor9wrhR3hPCKLcBe+vRdDZoa+7CSN8CLA/Dz33DnYPeJXPDninLjQ7EFl27woJRtblysq 4m8OTnEabv/nQr11QwvRkQxaoYRy09/3TSSZm5gf8UH60A102X9TU9+qAPeBCA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4T19733yFmz9yx; Thu, 28 Dec 2023 13:57:19 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 3BSDvJsX036227; Thu, 28 Dec 2023 13:57:19 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 3BSDvJ5D036224; Thu, 28 Dec 2023 13:57:19 GMT (envelope-from git) Date: Thu, 28 Dec 2023 13:57:19 GMT Message-Id: <202312281357.3BSDvJ5D036224@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Arthur Kiyanovski Subject: git: f97993ad7b9c - main - ena: Support customer metric with sysctl List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: akiyano X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: f97993ad7b9c9e4787bd198d11f348c271af513e Auto-Submitted: auto-generated The branch main has been updated by akiyano: URL: https://cgit.FreeBSD.org/src/commit/?id=f97993ad7b9c9e4787bd198d11f348c271af513e commit f97993ad7b9c9e4787bd198d11f348c271af513e Author: Osama Abboud AuthorDate: 2023-09-12 10:17:03 +0000 Commit: Arthur Kiyanovski CommitDate: 2023-12-28 13:56:24 +0000 ena: Support customer metric with sysctl This commit adds sysctl support for customer metrics. Different customer metrics can be found in the following sysctl node: sysctl dev.ena..customer_metrics Approved by: cperciva (mentor) MFC after: 2 weeks Sponsored by: Amazon, Inc. --- sys/dev/ena/ena.c | 49 +++++++++++++++++++++++++-- sys/dev/ena/ena.h | 1 + sys/dev/ena/ena_sysctl.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++- sys/dev/ena/ena_sysctl.h | 1 + 4 files changed, 135 insertions(+), 3 deletions(-) diff --git a/sys/dev/ena/ena.c b/sys/dev/ena/ena.c index f6357a43ea34..64c3d79f00db 100644 --- a/sys/dev/ena/ena.c +++ b/sys/dev/ena/ena.c @@ -166,6 +166,7 @@ static int ena_enable_msix_and_set_admin_interrupts(struct ena_adapter *); static void ena_update_on_link_change(void *, struct ena_admin_aenq_entry *); static void unimplemented_aenq_handler(void *, struct ena_admin_aenq_entry *); static int ena_copy_eni_metrics(struct ena_adapter *); +static int ena_copy_customer_metrics(struct ena_adapter *); static void ena_timer_service(void *); static char ena_version[] = ENA_DEVICE_NAME ENA_DRV_MODULE_NAME @@ -3305,6 +3306,25 @@ ena_copy_eni_metrics(struct ena_adapter *adapter) return (rc); } +static int +ena_copy_customer_metrics(struct ena_adapter *adapter) +{ + struct ena_com_dev *dev; + u32 supported_metrics_count; + int rc, len; + + dev = adapter->ena_dev; + + supported_metrics_count = ena_com_get_customer_metric_count(dev); + len = supported_metrics_count * sizeof(u64); + + /* Fill the data buffer */ + rc = ena_com_get_customer_metrics(adapter->ena_dev, + (char *)(adapter->customer_metrics_array), len); + + return (rc); +} + static void ena_timer_service(void *data) { @@ -3542,7 +3562,12 @@ ena_metrics_task(void *arg, int pending) struct ena_adapter *adapter = (struct ena_adapter *)arg; ENA_LOCK_LOCK(); - (void)ena_copy_eni_metrics(adapter); + + if (ena_com_get_cap(adapter->ena_dev, ENA_ADMIN_CUSTOMER_METRICS)) + (void)ena_copy_customer_metrics(adapter); + else if (ena_com_get_cap(adapter->ena_dev, ENA_ADMIN_ENI_STATS)) + (void)ena_copy_eni_metrics(adapter); + ENA_LOCK_UNLOCK(); } @@ -3756,6 +3781,18 @@ ena_attach(device_t pdev) /* initialize rings basic information */ ena_init_io_rings(adapter); + rc = ena_com_allocate_customer_metrics_buffer(ena_dev); + if (rc) { + ena_log(pdev, ERR, "Failed to allocate customer metrics buffer.\n"); + goto err_msix_free; + } + + rc = ena_sysctl_allocate_customer_metrics_buffer(adapter); + if (unlikely(rc)){ + ena_log(pdev, ERR, "Failed to allocate sysctl customer metrics buffer.\n"); + goto err_metrics_buffer_destroy; + } + /* Initialize statistics */ ena_alloc_counters((counter_u64_t *)&adapter->dev_stats, sizeof(struct ena_stats_dev)); @@ -3767,7 +3804,7 @@ ena_attach(device_t pdev) rc = ena_setup_ifnet(pdev, adapter, &get_feat_ctx); if (unlikely(rc != 0)) { ena_log(pdev, ERR, "Error with network interface setup\n"); - goto err_msix_free; + goto err_customer_metrics_alloc; } /* Initialize reset task queue */ @@ -3805,6 +3842,10 @@ ena_attach(device_t pdev) err_detach: ether_ifdetach(adapter->ifp); #endif /* DEV_NETMAP */ +err_customer_metrics_alloc: + free(adapter->customer_metrics_array, M_DEVBUF); +err_metrics_buffer_destroy: + ena_com_delete_customer_metrics_buffer(ena_dev); err_msix_free: ena_free_stats(adapter); ena_com_dev_reset(adapter->ena_dev, ENA_REGS_RESET_INIT_ERR); @@ -3903,6 +3944,10 @@ ena_detach(device_t pdev) ena_com_delete_host_info(ena_dev); + free(adapter->customer_metrics_array, M_DEVBUF); + + ena_com_delete_customer_metrics_buffer(ena_dev); + if_free(adapter->ifp); free(ena_dev->bus, M_DEVBUF); diff --git a/sys/dev/ena/ena.h b/sys/dev/ena/ena.h index fd0c6e3692c7..18988d719a18 100644 --- a/sys/dev/ena/ena.h +++ b/sys/dev/ena/ena.h @@ -487,6 +487,7 @@ struct ena_adapter { struct ena_stats_dev dev_stats; struct ena_hw_stats hw_stats; struct ena_admin_eni_stats eni_metrics; + uint64_t *customer_metrics_array; enum ena_regs_reset_reason_types reset_reason; }; diff --git a/sys/dev/ena/ena_sysctl.c b/sys/dev/ena/ena_sysctl.c index 572bccadca8c..82030a2a7539 100644 --- a/sys/dev/ena/ena_sysctl.c +++ b/sys/dev/ena/ena_sysctl.c @@ -37,6 +37,7 @@ static void ena_sysctl_add_wd(struct ena_adapter *); static void ena_sysctl_add_stats(struct ena_adapter *); static void ena_sysctl_add_eni_metrics(struct ena_adapter *); +static void ena_sysctl_add_customer_metrics(struct ena_adapter *); static void ena_sysctl_add_tuneables(struct ena_adapter *); static void ena_sysctl_add_irq_affinity(struct ena_adapter *); /* Kernel option RSS prevents manipulation of key hash and indirection table. */ @@ -58,6 +59,39 @@ static int ena_sysctl_rss_indir_table(SYSCTL_HANDLER_ARGS); #define ENA_METRICS_MAX_SAMPLE_INTERVAL 3600 #define ENA_HASH_KEY_MSG_SIZE (ENA_HASH_KEY_SIZE * 2 + 1) +#define SYSCTL_GSTRING_LEN 64 + +#define ENA_METRIC_ENI_ENTRY(stat, desc) { \ + .name = #stat, \ + .description = #desc, \ +} + +struct ena_hw_metrics { + char name[SYSCTL_GSTRING_LEN]; + char description[SYSCTL_GSTRING_LEN]; +}; + +static const struct ena_hw_metrics ena_hw_stats_strings[] = { + ENA_METRIC_ENI_ENTRY( + bw_in_allowance_exceeded, Inbound BW allowance exceeded), + ENA_METRIC_ENI_ENTRY( + bw_out_allowance_exceeded, Outbound BW allowance exceeded), + ENA_METRIC_ENI_ENTRY( + pps_allowance_exceeded, PPS allowance exceeded), + ENA_METRIC_ENI_ENTRY( + conntrack_allowance_exceeded, Connection tracking allowance exceeded), + ENA_METRIC_ENI_ENTRY( + linklocal_allowance_exceeded, Linklocal packet rate allowance), + ENA_METRIC_ENI_ENTRY( + conntrack_allowance_available, Number of available conntracks), +}; + +#ifndef ARRAY_SIZE +#define ARRAY_SIZE(a) (sizeof(a) / sizeof(a[0])) +#endif + +#define ENA_CUSTOMER_METRICS_ARRAY_SIZE ARRAY_SIZE(ena_hw_stats_strings) + static SYSCTL_NODE(_hw, OID_AUTO, ena, CTLFLAG_RD | CTLFLAG_MPSAFE, 0, "ENA driver parameters"); @@ -98,12 +132,29 @@ SYSCTL_BOOL(_hw_ena, OID_AUTO, force_large_llq_header, CTLFLAG_RDTUN, int ena_rss_table_size = ENA_RX_RSS_TABLE_SIZE; +int ena_sysctl_allocate_customer_metrics_buffer(struct ena_adapter *adapter) +{ + int rc = 0; + + adapter->customer_metrics_array = malloc((sizeof(u64) * ENA_CUSTOMER_METRICS_ARRAY_SIZE), + M_DEVBUF, M_NOWAIT | M_ZERO); + if (unlikely(adapter->customer_metrics_array == NULL)) + rc = ENOMEM; + + return rc; +} void ena_sysctl_add_nodes(struct ena_adapter *adapter) { + struct ena_com_dev *dev = adapter->ena_dev; + + if (ena_com_get_cap(dev, ENA_ADMIN_CUSTOMER_METRICS)) + ena_sysctl_add_customer_metrics(adapter); + else if (ena_com_get_cap(dev, ENA_ADMIN_ENI_STATS)) + ena_sysctl_add_eni_metrics(adapter); + ena_sysctl_add_wd(adapter); ena_sysctl_add_stats(adapter); - ena_sysctl_add_eni_metrics(adapter); ena_sysctl_add_tuneables(adapter); ena_sysctl_add_irq_affinity(adapter); #ifndef RSS @@ -328,6 +379,40 @@ ena_sysctl_add_stats(struct ena_adapter *adapter) &admin_stats->no_completion, 0, "Commands not completed"); } +static void +ena_sysctl_add_customer_metrics(struct ena_adapter *adapter) +{ + device_t dev; + struct ena_com_dev *ena_dev; + + struct sysctl_ctx_list *ctx; + struct sysctl_oid *tree; + struct sysctl_oid_list *child; + + struct sysctl_oid *customer_metric; + struct sysctl_oid_list *customer_list; + + int i; + + dev = adapter->pdev; + ena_dev = adapter->ena_dev; + + ctx = device_get_sysctl_ctx(dev); + tree = device_get_sysctl_tree(dev); + child = SYSCTL_CHILDREN(tree); + customer_metric = SYSCTL_ADD_NODE(ctx, child, OID_AUTO, "customer_metrics", + CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, "ENA's customer metrics"); + customer_list = SYSCTL_CHILDREN(customer_metric); + + for (i = 0; i < ENA_CUSTOMER_METRICS_ARRAY_SIZE; i++) { + if (ena_com_get_customer_metric_support(ena_dev, i)) { + SYSCTL_ADD_U64(ctx, customer_list, OID_AUTO, ena_hw_stats_strings[i].name, + CTLFLAG_RD, &adapter->customer_metrics_array[i], 0, + ena_hw_stats_strings[i].description); + } + } +} + static void ena_sysctl_add_eni_metrics(struct ena_adapter *adapter) { diff --git a/sys/dev/ena/ena_sysctl.h b/sys/dev/ena/ena_sysctl.h index 188c3a61a3d2..e9b4bfaae1cb 100644 --- a/sys/dev/ena/ena_sysctl.h +++ b/sys/dev/ena/ena_sysctl.h @@ -40,6 +40,7 @@ void ena_sysctl_add_nodes(struct ena_adapter *adapter); void ena_sysctl_update_queue_node_nb(struct ena_adapter *adapter, int old, int new); +int ena_sysctl_allocate_customer_metrics_buffer(struct ena_adapter *adapter); extern int ena_enable_9k_mbufs; #define ena_mbuf_sz (ena_enable_9k_mbufs ? MJUM9BYTES : MJUMPAGESIZE)