From nobody Mon Apr 18 10:41:46 2022 X-Original-To: dev-commits-ports-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5C1EFCFC402; Mon, 18 Apr 2022 10:41:46 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Khk56293Qz3MGy; Mon, 18 Apr 2022 10:41:46 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1650278506; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=SZSXVaSDkP9t+YPf2Zn5aWNO89f5mDTr6o6wOEityY8=; b=NYXzId7RdwUmnoKN3vSzHoDA7ku452manZCn5xbikkcnOEqBykU1fnuiIRee5sN6WmuZCh eBWUMPGyDxrQ8goNE0q74f3T4Jw/vrrm/FIlKTZ0gc3MAT2c90kz07orZAC7+mMIs3JzgV cQQ6vVdg6jNt+HD9gAfc3cn0+kmTQXapz/b/OywjJU4J/ImNML8eYtiIzBVfNSVo/wnMN7 7OrSloC5lfagltGwYdV2nvC1ZvVmnhxYTuJUU8IIWyXWg92mjIEvFg0pTUB6XdrvfAPmwV wWEMYZgPKpdcTrfXf2R6umgu/OuFbMmAaS27YKyY1Uy1w5xr6vFYP3t+tdqstw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 256B311240; Mon, 18 Apr 2022 10:41:46 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 23IAfkEZ008450; Mon, 18 Apr 2022 10:41:46 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 23IAfkvg008449; Mon, 18 Apr 2022 10:41:46 GMT (envelope-from git) Date: Mon, 18 Apr 2022 10:41:46 GMT Message-Id: <202204181041.23IAfkvg008449@gitrepo.freebsd.org> To: ports-committers@FreeBSD.org, dev-commits-ports-all@FreeBSD.org, dev-commits-ports-main@FreeBSD.org From: Dmitry Marakasov Subject: git: b731fff69c0f - main - Mk/Uses: Add pytest.mk List-Id: Commits to the main branch of the FreeBSD ports repository List-Archive: https://lists.freebsd.org/archives/dev-commits-ports-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-ports-main@freebsd.org X-BeenThere: dev-commits-ports-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: amdmi3 X-Git-Repository: ports X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: b731fff69c0ffd79f80f5d1d575e016337f3b920 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1650278506; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=SZSXVaSDkP9t+YPf2Zn5aWNO89f5mDTr6o6wOEityY8=; b=DajRD58zzNb1QFHn9IAuw6ID7ftUJCFVWP59BzWwhp3BVrGKm2tUdsVmuIjf6bVMiySO/S 1idp60dx6vIbxuV6KjiBh7fyMn3bjVLMLG2xyCKfypFQK52/fB/2PXx0Lc/g9inT9t73wa lMAMiAgQEUc9MMc1SFfxDhxALpe/CDy8kv/YPzlLmhG2+ujvGYmhEBsqNYSdMVyPyGH/tK ZsJDIEAmYotP5Kb2lrhDJMZJYzNpKTp+um8r2t1baHhodrm+MQW10lUEwxIyGUWqCvXyzw ZL0s0dTk8jKEjPhJ5dGfaL/Abeld5JOvuB6uwwu/T/1VOfnZZrtuy//pM8IHrg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1650278506; a=rsa-sha256; cv=none; b=YGts/mBOV2ANnHrro5JtQFQvPryYRWoSTt50JhmsFKL5h07E5CC5lcLW/8kFLq9tOoYQll TemcBXE2o0IpsRJ73WF58qT7jTxXiL49myKL4DiMj5sWtemzoOPuhyX2VL3e/BZiXdbEaa 2IsUSzpIUOOo+qBMWL7qbBUkxl3cG7ZQTOb+hVfLuZ5jnx/TkaB/uh4EccZ81M/rTzJjci 8vOHMeqaCWP+xlyGmfWxLNC0izPkMWMDJvv3ZRnOb2/tAnBy9o2CeUmJz6VserYq1cdqA0 cjdrx2cfuYDgcM6GZ+lP8VysQKzyKuFN1W++wMiz8ZgjoXqhekDZh/kKVBhDaQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by amdmi3: URL: https://cgit.FreeBSD.org/ports/commit/?id=b731fff69c0ffd79f80f5d1d575e016337f3b920 commit b731fff69c0ffd79f80f5d1d575e016337f3b920 Author: Dmitry Marakasov AuthorDate: 2022-04-15 14:17:30 +0000 Commit: Dmitry Marakasov CommitDate: 2022-04-18 10:32:16 +0000 Mk/Uses: Add pytest.mk Add USES=pytest, a standardized pytest entry point. This knob adds dependency on pytest and defines do-test target which calls it "the right way" (respecting TEST_ENV, with proper verbosity, additional info on skipped tests, and resetting addopts which may be set by upstream and introduce unwanted options). This allows to remove duplicate (and in many cases not completely correct) do-test targets from a lot of pytest consumer ports. Additionally, it adds PYTEST_IGNORED_TESTS (to list tests which fail and are expected to fail) and PYTEST_BROKEN_TESTS (to list tests which fail and needs to be fixed). Both knobs lead to listed tests being skipped, which is believed to be better than always failing `test` target, allowing to monitor status of other tests and better document known failures. Possible room for improvement: - Support for pytest plugins (add dependencies and disallow pytest from using unlisted plugins which often cause breakages) - Treat PYTEST_BROKEN_TESTS as xfail to catch the cases when tests are fixed. Both items seem to require support on pytest side though. Reviewed by: tcberner Differential Revision: https://reviews.freebsd.org/D34693 --- CHANGES | 13 +++++++++++ Mk/Uses/pytest.mk | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 82 insertions(+) diff --git a/CHANGES b/CHANGES index bb1fdcf4d997..e96e1b0d4a0d 100644 --- a/CHANGES +++ b/CHANGES @@ -10,6 +10,19 @@ in the release notes and/or placed into UPDATING. All ports committers are allowed to commit to this file. +20220415: +AUTHOR: amdmi3@FreeBSD.org + + A new USES has been added to handle testing with pytest. + + USES= pytest + + Introduces dependency on pytest and adds do-test target which calls + pytest with the right environment and arguments. + + Additionally, PYTEST_{IGNORED,BROKEN}_TESTS knobs are provided for + skipping failing tests in a convenient and documented way. + 20220218: AUTHOR: jrm@FreeBSD.org diff --git a/Mk/Uses/pytest.mk b/Mk/Uses/pytest.mk new file mode 100644 index 000000000000..62e435270cbd --- /dev/null +++ b/Mk/Uses/pytest.mk @@ -0,0 +1,69 @@ +# handle testing with pytest +# +# Feature: pytest +# Usage: USES=pytest[:4] +# +# It implies USES=python:test automatically if no USES=python has been +# specified yet +# +# It provides the following additional variables to set by the ports: +# +# PYTEST_ARGS additional args to pytest (defaults to empty). +# PYTEST_IGNORED_TESTS lists of `pytest -k` patterns of tests to ignore +# (defaults to empty). For tests which are not +# expected to pass, such as ones requiring a database +# access. +# PYTEST_BROKEN_TESTS lists of `pytest -k` patterns of tests to ignore +# (defaults to empty). For broken tests which require +# fixing. +# +# The following variables may be set by the user: +# +# PYTEST_ENABLE_IGNORED_TESTS enable tests which are otherwise ignored by +# PYTEST_IGNORED_TESTS. +# PYTEST_ENABLE_BROKEN_TESTS enable tests which are otherwise ignored by +# PYTEST_BROKEN_TESTS. +# PYTEST_ENABLE_ALL_TESTS enable tests which are otherwise ignored by +# PYTEST_IGNORED_TESTS and PYTEST_BROKEN_TESTS. +# +# MAINTAINER: amdmi3@FreeBSD.org + +.if !defined(_INCLUDE_USES_PYTEST_MK) +_INCLUDE_USES_PYTEST_MK= yes + +. if !${USES:Mpython*} +python_ARGS= test +. include "${USESDIR}/python.mk" +. endif + +. if empty(pytest_ARGS) +TEST_DEPENDS+= ${PYTHON_PKGNAMEPREFIX}pytest>=0:devel/py-pytest@${PY_FLAVOR} +. elif ${pytest_ARGS} == "4" +TEST_DEPENDS+= ${PYTHON_PKGNAMEPREFIX}pytest4>=0:devel/py-pytest4@${PY_FLAVOR} +. else +IGNORE= Incorrect 'USES+=pytest:${pytest_ARGS}' expecting 'USES+=pytest[:4]' +. endif + +PYTEST_IGNORED_TESTS?= # empty +PYTEST_BROKEN_TESTS?= # empty +PYTEST_ARGS?= # empty + +_PYTEST_ALL_IGNORED_TESTS?= # empty +. if !defined(PYTEST_ENABLE_IGNORED_TESTS) && !defined(PYTEST_ENABLE_ALL_TESTS) +_PYTEST_ALL_IGNORED_TESTS+= ${PYTEST_IGNORED_TESTS} +. endif +. if !defined(PYTEST_ENABLE_BROKEN_TESTS) && !defined(PYTEST_ENABLE_ALL_TESTS) +_PYTEST_ALL_IGNORED_TESTS+= ${PYTEST_BROKEN_TESTS} +. endif + +_PYTEST_FILTER_EXPRESSION= ${_PYTEST_ALL_IGNORED_TESTS:C/^(.)/and not \1/:tW:C/^and //} + +. if !target(do-test) +do-test: + @cd ${TEST_WRKSRC} && ${SETENV} ${TEST_ENV} ${PYTHON_CMD} -m pytest \ + -k '${_PYTEST_FILTER_EXPRESSION}' \ + -v -rs -o addopts= \ + ${PYTEST_ARGS} +. endif + +.endif