From nobody Sat Oct 05 11:34:33 2024 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XLNc93lcXz5YBQR; Sat, 05 Oct 2024 11:34:33 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XLNc93MTtz4rqg; Sat, 5 Oct 2024 11:34:33 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1728128073; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=fPQ4VOsjfThRmeG4fAFt/7SNWnq5DmFJctDEpN8cEvg=; b=nnK5jE0Anu7LOiyKtD1NkhVGgO4W/fnHy83zEtirXIvSh7mJVTWmf/+psA/M7vLQliUJe6 xfNqK0LKi1kjDVu48hmZb86l6Tu0uoRNL3NUhZwaICG6UrUPY1OKkzLB3zYLv0aI8dlM9Y psWYbM/tWSQaI/COFkcb0NS7lG93uWBAfyaSJXfgnWS+Sdvbq2uffslDiUOLUWvWEJ+kLq m2BVVHx+vR0Fey4JUELEakooryyIYnKwak01Y5L0fCHpGx4usX7TYTsx9v8pICD+NMtzxQ vTe2ywOfSvMw9Eii5ehfkucG2pF11PiYWOIZp4nXgh3lC7HCJFmaBlndv6e0vg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1728128073; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=fPQ4VOsjfThRmeG4fAFt/7SNWnq5DmFJctDEpN8cEvg=; b=UpRaZKOAsPgnnQPlqKT6KTX6Oq10AoJn338fDE9w0/GCUPzQ9GfFJWwwP8KQVOrMYVZvgN dH2rdnWOt1YBYv4OWy5ypDX/GaqiW/M4AmfvJ6jyD7cFjY8SmFuv8dY6FvhjoQ0pk6bUjZ RtWGpBMVXwu74m74mhskOWHCtSOEWAzwIRIPAbC274Llb4aEd5hVqnPp8bfipfO/iS7Z3P bsNc0VBp4hiqxcEE6riqUaOIwO7CwqKl4Y7GttIisCymXDISBosZ2LxIpS4qpj45vRplbF Mn78qDj8YkGqjKA9FOLjV9H5YxRiEj9uAgiHB3B1FUal70P/IaDgZMAtwAmRAA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1728128073; a=rsa-sha256; cv=none; b=gKHFrSEZ5/QmcFQVMAdoM2r4YyxagXTLcOheaXgRZ7e4RbbQL68OFJ7E5q5nAZoITwx3Im HQKPlvy8LPCneElMHqL+WL2r5kPHraJpx+qH5jKoubDOZdTvm4f/zMkRrSFXJZMigp2bck WHG0aFcJsTycfrKEir9PN6pj46eN0e9Aqv2so/YRnVs6KQSn8PRUeMzjUn3B1yCMXhuw6j ktPaAKDFhhA+BNrpOoL6PfgILkogdH3auglV1L2D76QD2lm90WQ/zB8SIx5su3/KPOhuYz pTe48M9W4WZzRqGykaAwv7vvYnvL+lTuOxGZ+iHm9FwgpiOVH5nxK+dhbXMaLQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XLNc92qkjzH5F; Sat, 5 Oct 2024 11:34:33 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 495BYXY7047818; Sat, 5 Oct 2024 11:34:33 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 495BYXar047815; Sat, 5 Oct 2024 11:34:33 GMT (envelope-from git) Date: Sat, 5 Oct 2024 11:34:33 GMT Message-Id: <202410051134.495BYXar047815@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Igor Ostapenko Subject: git: 99689201a1eb - main - kyua: Do not count skipped as passed in test cmd List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: igoro X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 99689201a1eb7bcd503987c5220b79330dc058b7 Auto-Submitted: auto-generated The branch main has been updated by igoro: URL: https://cgit.FreeBSD.org/src/commit/?id=99689201a1eb7bcd503987c5220b79330dc058b7 commit 99689201a1eb7bcd503987c5220b79330dc058b7 Author: Igor Ostapenko AuthorDate: 2024-10-05 10:19:31 +0000 Commit: Igor Ostapenko CommitDate: 2024-10-05 10:22:22 +0000 kyua: Do not count skipped as passed in test cmd It changes output of 'kyua test' CLI command only. Hence, other outputs like junit are kept intact for CI and other use cases. It's meant to improve UX of attended use cases. The issue is that the following can be tricky to interpret: 222/222 passed (0 failed) It can be read as all tests are passed, but it might be a summary line of all tests skipped due to some requirement is not met. It's reworked to easily distinguish such cases: 222/222 passed (0 broken, 0 failed, 0 skipped) 0/222 passed (0 broken, 0 failed, 222 skipped) The overall formula is: / (
) Suggested by: kp Reviewed by: ngie, markj Approved by: markj (mentor) Differential Revision: https://reviews.freebsd.org/D46653 --- contrib/kyua/cli/cmd_test.cpp | 50 +++++++++++++++++++++---------- contrib/kyua/integration/cmd_test_test.sh | 42 +++++++++++++------------- contrib/kyua/model/test_result.cpp | 48 ++++++++++++++++++++++------- contrib/kyua/model/test_result.hpp | 15 ++++++++++ 4 files changed, 107 insertions(+), 48 deletions(-) diff --git a/contrib/kyua/cli/cmd_test.cpp b/contrib/kyua/cli/cmd_test.cpp index cfaeec9b74cc..b7afa6d11547 100644 --- a/contrib/kyua/cli/cmd_test.cpp +++ b/contrib/kyua/cli/cmd_test.cpp @@ -64,11 +64,8 @@ class print_hooks : public drivers::run_tests::base_hooks { bool _parallel; public: - /// The amount of positive test results found so far. - unsigned long good_count; - - /// The amount of negative test results found so far. - unsigned long bad_count; + /// The amount of test results per type. + std::map type_count; /// Constructor for the hooks. /// @@ -76,10 +73,10 @@ public: /// \param parallel_ True if we are executing more than one test at once. print_hooks(cmdline::ui* ui_, const bool parallel_) : _ui(ui_), - _parallel(parallel_), - good_count(0), - bad_count(0) + _parallel(parallel_) { + for (const auto& pair : model::test_result_types) + type_count[pair.first] = 0; } /// Called when the processing of a test case begins. @@ -116,10 +113,8 @@ public: } _ui->out(F("%s [%s]") % cli::format_result(result) % cli::format_delta(duration)); - if (result.good()) - good_count++; - else - bad_count++; + + type_count[result.type()]++; } }; @@ -159,8 +154,21 @@ cmd_test::run(cmdline::ui* ui, const cmdline::parsed_cmdline& cmdline, kyuafile_path(cmdline), build_root_path(cmdline), results.second, parse_filters(cmdline.arguments()), user_config, hooks); + unsigned long total = 0; + unsigned long good = 0; + unsigned long bad = 0; + for (const auto& pair : model::test_result_types) { + const auto& type = pair.second; + const auto count = hooks.type_count[type.id]; + total += count; + if (type.is_run && type.is_good) + good += count; + if (!type.is_good) + bad += count; + } + int exit_code; - if (hooks.good_count > 0 || hooks.bad_count > 0) { + if (total > 0) { ui->out(""); if (!results.first.empty()) { ui->out(F("Results file id is %s") % results.first); @@ -168,10 +176,20 @@ cmd_test::run(cmdline::ui* ui, const cmdline::parsed_cmdline& cmdline, ui->out(F("Results saved to %s") % results.second); ui->out(""); - ui->out(F("%s/%s passed (%s failed)") % hooks.good_count % - (hooks.good_count + hooks.bad_count) % hooks.bad_count); + ui->out(F("%s/%s passed (") % good % total, false); + const auto& types = model::test_result_types; + for (auto it = types.begin(); it != types.end(); it++) { + const auto& type = it->second; + if (!type.is_run || !type.is_good) { + if (it != types.begin()) + ui->out(", ", false); + ui->out(F("%s %s") % hooks.type_count[type.id] % type.name, + false); + } + } + ui->out(")"); - exit_code = (hooks.bad_count == 0 ? EXIT_SUCCESS : EXIT_FAILURE); + exit_code = (bad == 0 ? EXIT_SUCCESS : EXIT_FAILURE); } else { // TODO(jmmv): Delete created empty file; it's useless! if (!results.first.empty()) { diff --git a/contrib/kyua/integration/cmd_test_test.sh b/contrib/kyua/integration/cmd_test_test.sh index bc8c62daf223..f0862cf29582 100644 --- a/contrib/kyua/integration/cmd_test_test.sh +++ b/contrib/kyua/integration/cmd_test_test.sh @@ -44,7 +44,7 @@ simple_all_pass:skip -> skipped: The reason for skipping is this [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -2/2 passed (0 failed) +1/2 passed (0 broken, 0 failed, 1 skipped) EOF utils_cp_helper simple_all_pass . @@ -69,7 +69,7 @@ simple_some_fail:pass -> passed [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -1/2 passed (1 failed) +1/2 passed (0 broken, 1 failed, 0 skipped) EOF utils_cp_helper simple_some_fail . @@ -102,7 +102,7 @@ third:skip -> skipped: The reason for skipping is this [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -7/7 passed (0 failed) +3/7 passed (0 broken, 0 failed, 4 skipped) EOF utils_cp_helper simple_all_pass first @@ -138,7 +138,7 @@ third:skip -> skipped: The reason for skipping is this [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -4/7 passed (3 failed) +3/7 passed (0 broken, 3 failed, 1 skipped) EOF utils_cp_helper simple_some_fail first @@ -172,7 +172,7 @@ expect_all_pass:timeout -> expected_failure: This times out [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -5/5 passed (0 failed) +5/5 passed (0 broken, 0 failed, 0 skipped) EOF # CHECK_STYLE_ENABLE @@ -203,7 +203,7 @@ expect_some_fail:timeout -> failed: Test case was expected to hang but it cont Results file id is $(utils_results_id) Results saved to $(utils_results_file) -1/6 passed (5 failed) +1/6 passed (0 broken, 5 failed, 0 skipped) EOF # CHECK_STYLE_ENABLE @@ -231,7 +231,7 @@ bogus_test_cases:pass -> passed [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -1/3 passed (2 failed) +1/3 passed (2 broken, 0 failed, 0 skipped) EOF # CHECK_STYLE_ENABLE @@ -270,7 +270,7 @@ subdir/simple_some_fail:pass -> passed [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -3/4 passed (1 failed) +2/4 passed (0 broken, 1 failed, 1 skipped) EOF atf_check -s exit:1 -o file:expout -e empty kyua test } @@ -302,7 +302,7 @@ subdir/simple_all_pass:skip -> skipped: The reason for skipping is this [S.UU Results file id is $(utils_results_id) Results saved to $(utils_results_file) -2/2 passed (0 failed) +1/2 passed (0 broken, 0 failed, 1 skipped) EOF # CHECK_STYLE_ENABLE atf_check -s exit:0 -o file:expout -e empty kyua test subdir @@ -328,7 +328,7 @@ first:skip -> skipped: The reason for skipping is this [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -1/1 passed (0 failed) +0/1 passed (0 broken, 0 failed, 1 skipped) EOF atf_check -s exit:0 -o file:expout -e empty kyua test first:skip } @@ -354,7 +354,7 @@ second:pass -> passed [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -1/2 passed (1 failed) +1/2 passed (0 broken, 1 failed, 0 skipped) EOF atf_check -s exit:1 -o file:expout -e empty kyua test second } @@ -402,7 +402,7 @@ subdir/second:pass -> passed [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -2/3 passed (1 failed) +2/3 passed (0 broken, 1 failed, 0 skipped) EOF atf_check -s exit:1 -o file:expout -e empty kyua test subdir first:pass } @@ -470,7 +470,7 @@ third:pass -> passed [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -3/4 passed (1 failed) +2/4 passed (0 broken, 1 failed, 1 skipped) EOF cat >experr < failed: This fails on purpose [S.UUUs] Results file id is $(utils_results_id root) Results saved to $(utils_results_file root) -2/3 passed (1 failed) +1/3 passed (0 broken, 1 failed, 1 skipped) EOF atf_check -s exit:1 -o file:expout -e empty kyua test \ -k "$(pwd)/root/Kyuafile" first subdir/fourth:fail @@ -542,7 +542,7 @@ first:skip -> skipped: The reason for skipping is this [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -2/2 passed (0 failed) +1/2 passed (0 broken, 0 failed, 1 skipped) EOF CREATE_COOKIE="$(pwd)/cookie"; export CREATE_COOKIE atf_check -s exit:0 -o file:expout -e empty kyua test first @@ -612,7 +612,7 @@ some-program:pass -> passed [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -1/2 passed (1 failed) +1/2 passed (0 broken, 1 failed, 0 skipped) EOF atf_check -s exit:1 -o file:expout -e empty kyua test @@ -710,7 +710,7 @@ subdir/third:skip -> skipped: The reason for skipping is this [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -6/6 passed (0 failed) +3/6 passed (0 broken, 0 failed, 3 skipped) EOF mkdir build @@ -745,7 +745,7 @@ sometest:skip -> skipped: The reason for skipping is this [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -2/2 passed (0 failed) +1/2 passed (0 broken, 0 failed, 1 skipped) EOF atf_check -s exit:0 -o file:expout -e empty kyua test -k myfile atf_check -s exit:0 -o file:expout -e empty kyua test --kyuafile=myfile @@ -774,7 +774,7 @@ sometest:skip -> skipped: The reason for skipping is this [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -2/2 passed (0 failed) +1/2 passed (0 broken, 0 failed, 1 skipped) EOF atf_check -s exit:0 -o file:expout -e empty kyua test -k myfile sometest cat >expout < skipped: The reason for skipping is this [S.UUUs] Results file id is $(utils_results_id) Results saved to $(utils_results_file) -2/2 passed (0 failed) +1/2 passed (0 broken, 0 failed, 1 skipped) EOF atf_check -s exit:0 -o file:expout -e empty kyua test --kyuafile=myfile \ sometest @@ -987,7 +987,7 @@ non_executable:__test_cases_list__ -> broken: Permission denied to run test pr Results file id is $(utils_results_id) Results saved to $(utils_results_file) -0/2 passed (2 failed) +0/2 passed (2 broken, 0 failed, 0 skipped) EOF # CHECK_STYLE_ENABLE atf_check -s exit:1 -o file:expout -e empty kyua test diff --git a/contrib/kyua/model/test_result.cpp b/contrib/kyua/model/test_result.cpp index 7392e77f5561..f3e1b66f260b 100644 --- a/contrib/kyua/model/test_result.cpp +++ b/contrib/kyua/model/test_result.cpp @@ -35,6 +35,42 @@ namespace text = utils::text; +const std::map + model::test_result_types = +{ + { test_result_broken, + { .id = test_result_broken, + .name = "broken", + .is_run = true, + .is_good = false, } }, + + { test_result_expected_failure, + { .id = test_result_expected_failure, + .name = "xfail", + .is_run = true, + .is_good = true, } }, + + { test_result_failed, + { .id = test_result_failed, + .name = "failed", + .is_run = true, + .is_good = false, } }, + + { test_result_passed, + { .id = test_result_passed, + .name = "passed", + .is_run = true, + .is_good = true, } }, + + { test_result_skipped, + { .id = test_result_skipped, + .name = "skipped", + .is_run = false, + .is_good = true, } }, +}; + + /// Constructs a base result. /// /// \param type_ The type of the result. @@ -74,17 +110,7 @@ model::test_result::reason(void) const bool model::test_result::good(void) const { - switch (_type) { - case test_result_expected_failure: - case test_result_passed: - case test_result_skipped: - return true; - - case test_result_broken: - case test_result_failed: - return false; - } - UNREACHABLE; + return test_result_types.at(_type).is_good; } diff --git a/contrib/kyua/model/test_result.hpp b/contrib/kyua/model/test_result.hpp index b9c439ce789a..33c76124550f 100644 --- a/contrib/kyua/model/test_result.hpp +++ b/contrib/kyua/model/test_result.hpp @@ -34,12 +34,27 @@ #include "model/test_result_fwd.hpp" +#include #include #include namespace model { +/// Test result type metadata. +struct test_result_type_desc { + enum test_result_type id; + std::string name; + bool is_run; + bool is_good; +}; + + +/// Description of each test result type. +extern const std::map test_result_types; + + /// Representation of a single test result. /// /// A test result is a simple pair of (type, reason). The type indicates the