svn commit: r313790 - in stable/10: . share/man/man7 share/mk
Ngie Cooper
ngie at FreeBSD.org
Thu Feb 16 05:14:09 UTC 2017
Author: ngie
Date: Thu Feb 16 05:14:07 2017
New Revision: 313790
URL: https://svnweb.freebsd.org/changeset/base/313790
Log:
MFC r285119,r292502,r295380:
r285119 (by jmmv):
Add support for TEST_METADATA
Allow Makefiles to define generic metadata settings that apply to all test
programs defined by a Makefile. The generic TEST_METADATA variable extends
the per-test program settings already supported via TEST_METADATA.<program>.
This feature will be useful to easily apply some settings to all programs
in a directory. In particular, Kyua 0.12 will support parallel execution
of test programs and a bunch of them will need to be tagged as is_exclusive
to indicate that they cannot be run in parallel with anything else due to
their side-effects. It will be reasonable to set this setting on whole
directories.
r292502:
Always expose LOCALBASE, not just when CROSS_TOOLCHAIN is defined
Instead of using which(1) to look for doxygen, look for it in <LOCALBASE>/bin .
$PATH gets mangled by make buildenv, etc so it's better to just be explicit
about the path if someone uses that for instance.
r295380:
Simplify running the FreeBSD test suite
Replace `make regress` (legacy test make target) and `make test` (incomplete
test make target added with the FreeBSD test suite) with make check as it's
consistent with other open source projects.
`make check` defaults to running tests from `.OBJDIR`, but can be overridden
with the `CHECKDIR` variable.
Add `make checkworld` target to simplify running the FreeBSD test suite from
`TESTSBASE` (i.e. the top-level tests directory), similar to buildworld.
Document `make check` and `make checkworld` in build(7).
Other minor changes:
- Rename intermediate file (`Kyuafile.auto`) to `Kyuafile` to simplify
`make check`.
- Remove terse warnings attached to `beforetest`/`aftertest`.
- Add kyua binary check to check target in suite.test.mk; error out if it's
not found
The MFC is [partly] contingent on other build related changes being MFCed.
X-MFC to: stable/10
Relnotes: yes
Modified:
stable/10/Makefile
stable/10/Makefile.inc1
stable/10/share/man/man7/build.7
stable/10/share/mk/bsd.README
stable/10/share/mk/bsd.subdir.mk
stable/10/share/mk/bsd.sys.mk
stable/10/share/mk/bsd.test.mk
stable/10/share/mk/suite.test.mk
Directory Properties:
stable/10/ (props changed)
Modified: stable/10/Makefile
==============================================================================
--- stable/10/Makefile Thu Feb 16 04:23:45 2017 (r313789)
+++ stable/10/Makefile Thu Feb 16 05:14:07 2017 (r313790)
@@ -21,6 +21,7 @@
# kernel-toolchains - Build kernel-toolchain for all universe targets.
# doxygen - Build API documentation of the kernel, needs doxygen.
# update - Convenient way to update your source tree(s).
+# checkworld - Run test suite on installed world.
# check-old - List obsolete directories/files/libraries.
# check-old-dirs - List obsolete directories.
# check-old-files - List obsolete files.
@@ -99,8 +100,8 @@
# For more information, see the build(7) manual page.
#
TGTS= all all-man buildenv buildenvvars buildkernel buildworld \
- check-old check-old-dirs check-old-files check-old-libs \
- checkdpadd clean cleandepend cleandir \
+ check check-old check-old-dirs check-old-files check-old-libs \
+ checkdpadd checkworld clean cleandepend cleandir \
delete-old delete-old-dirs delete-old-files delete-old-libs \
depend distribute distributekernel distributekernel.debug \
distributeworld distrib-dirs distribution doxygen \
@@ -108,7 +109,7 @@ TGTS= all all-man buildenv buildenvvars
installkernel.debug packagekernel packageworld \
reinstallkernel reinstallkernel.debug \
installworld kernel-toolchain libraries lint maninstall \
- obj objlink regress rerelease showconfig tags toolchain update \
+ obj objlink rerelease showconfig tags toolchain update \
_worldtmp _legacy _bootstrap-tools _cleanobj _obj \
_build-tools _cross-tools _includes _libraries _depend \
build32 builddtb distribute32 install32 xdev xdev-build xdev-install \
@@ -374,6 +375,10 @@ make bmake: .PHONY
${MMAKE} all; \
${MMAKE} install DESTDIR=${MYMAKE:H} BINDIR=
+regress: .PHONY
+ @echo "'make regress' has been renamed 'make check'" | /usr/bin/fmt
+ @false
+
tinderbox toolchains kernel-toolchains: upgrade_checks
tinderbox:
Modified: stable/10/Makefile.inc1
==============================================================================
--- stable/10/Makefile.inc1 Thu Feb 16 04:23:45 2017 (r313789)
+++ stable/10/Makefile.inc1 Thu Feb 16 05:14:07 2017 (r313790)
@@ -35,6 +35,7 @@
# The intended user-driven targets are:
# buildworld - rebuild *everything*, including glue to help do upgrades
# installworld- install everything built by "buildworld"
+# checkworld - run test suite on installed world
# doxygen - build API documentation of the kernel
# update - convenient way to update your source tree (eg: svn/svnup)
#
@@ -50,6 +51,8 @@
.include <bsd.arch.inc.mk>
.include <bsd.compiler.mk>
+LOCALBASE?= /usr/local
+
# We must do share/info early so that installation of info `dir'
# entries works correctly. Do it first since it is less likely to
# grow dependencies on include and lib than vice versa.
@@ -1190,12 +1193,26 @@ packagekernel:
.endif
#
+#
+# checkworld
+#
+# Run test suite on installed world.
+#
+checkworld: .PHONY
+ @if [ ! -x ${LOCALBASE}/bin/kyua ]; then \
+ echo "You need kyua (devel/kyua) to run the test suite." | /usr/bin/fmt; \
+ exit 1; \
+ fi
+ ${_+_}${LOCALBASE}/bin/kyua test -k ${TESTSBASE}/Kyuafile
+
+#
+#
# doxygen
#
# Build the API documentation with doxygen
#
doxygen: .PHONY
- @if [ ! -x `/usr/bin/which doxygen` ]; then \
+ @if [ ! -x ${LOCALBASE}/bin/doxygen ]; then \
echo "You need doxygen (devel/doxygen) to generate the API documentation of the kernel." | /usr/bin/fmt; \
exit 1; \
fi
Modified: stable/10/share/man/man7/build.7
==============================================================================
--- stable/10/share/man/man7/build.7 Thu Feb 16 04:23:45 2017 (r313789)
+++ stable/10/share/man/man7/build.7 Thu Feb 16 05:14:07 2017 (r313790)
@@ -107,6 +107,16 @@ section below, and by the variables docu
The following list provides the names and actions for the targets
supported by the build system:
.Bl -tag -width ".Cm cleandepend"
+.It Cm check
+Run tests for a given subdirectory.
+The default directory used is
+.Pa ${.OBJDIR} ,
+but the check directory can be changed with
+.Pa ${CHECKDIR} .
+.It Cm checkworld
+Run the
+.Fx
+test suite on installed world.
.It Cm clean
Remove any files created during the build process.
.It Cm cleandepend
@@ -653,6 +663,7 @@ make TARGET=sparc64 DESTDIR=/clients/spa
.Xr mergemaster 8 ,
.Xr portsnap 8 ,
.Xr reboot 8 ,
-.Xr shutdown 8
+.Xr shutdown 8 ,
+.Xr tests 7
.Sh AUTHORS
.An Mike W. Meyer Aq mwm at mired.org .
Modified: stable/10/share/mk/bsd.README
==============================================================================
--- stable/10/share/mk/bsd.README Thu Feb 16 04:23:45 2017 (r313789)
+++ stable/10/share/mk/bsd.README Thu Feb 16 05:14:07 2017 (r313790)
@@ -492,6 +492,17 @@ It has seven targets:
all:
build the test programs.
+ check:
+ runs the test programs from CHECKDIR with kyua test.
+
+ The beforecheck and aftercheck targets will be invoked, if
+ defined, to execute commands before and after the realcheck
+ target has been executed, respectively.
+
+ The devel/kyua package must be installed before invoking this
+ target.
+
+ See CHECKDIR for more details.
clean:
remove the test programs and any object files.
cleandir:
@@ -510,12 +521,6 @@ It has seven targets:
run lint on the source files.
tags:
create a tags file for the source files.
- test:
- runs the test programs from the object directory; if the
- Makefile does not itself define the target test, the
- targets beforetest and aftertest may also be used to
- cause actions immediately before and after the test
- target is executed.
It sets/uses the following variables, among many others:
@@ -529,6 +534,10 @@ TESTSDIR Path to the installed tests. M
${TESTSBASE}/${RELDIR:H} , e.g. /usr/tests/bin/ls when
included from bin/ls/tests .
+CHECKDIR The directory that 'make check' executes tests from.
+
+ The value of CHECKDIR defaults to .OBJDIR.
+
KYUAFILE If 'auto' (the default), generate a Kyuafile out of the
test programs defined in the Makefile. If 'yes', then a
manually-crafted Kyuafile must be supplied with the
Modified: stable/10/share/mk/bsd.subdir.mk
==============================================================================
--- stable/10/share/mk/bsd.subdir.mk Thu Feb 16 04:23:45 2017 (r313789)
+++ stable/10/share/mk/bsd.subdir.mk Thu Feb 16 05:14:07 2017 (r313790)
@@ -24,9 +24,9 @@
# This is a variant of install, which will
# put the stuff into the right "distribution".
#
-# afterinstall, all, all-man, beforeinstall, checkdpadd, clean,
+# afterinstall, all, all-man, beforeinstall, check, checkdpadd, clean,
# cleandepend, cleandir, cleanilinks depend, install, lint,
-# maninstall, manlint, obj, objlink, realinstall, regress, tags
+# maninstall, manlint, obj, objlink, realinstall, tags
#
.if !target(__<bsd.subdir.mk>__)
@@ -70,9 +70,9 @@ ${SUBDIR:N.WAIT}: .PHONY .MAKE
# Work around parsing of .if nested in .for by putting .WAIT string into a var.
__wait= .WAIT
-.for __target in all all-man checkdpadd clean cleandepend cleandir \
+.for __target in all all-man check checkdpadd clean cleandepend cleandir \
cleanilinks depend distribute lint maninstall manlint obj objlink \
- realinstall regress tags ${SUBDIR_TARGETS}
+ realinstall tags ${SUBDIR_TARGETS}
.ifdef SUBDIR_PARALLEL
__subdir_targets=
.for __dir in ${SUBDIR}
Modified: stable/10/share/mk/bsd.sys.mk
==============================================================================
--- stable/10/share/mk/bsd.sys.mk Thu Feb 16 04:23:45 2017 (r313789)
+++ stable/10/share/mk/bsd.sys.mk Thu Feb 16 05:14:07 2017 (r313790)
@@ -158,11 +158,11 @@ CFLAGS+= ${CWARNFLAGS} ${CWARNFLAGS.${.I
# or expect to ever be up-to-date.
PHONY_NOTMAIN = afterdepend afterinstall all beforedepend beforeinstall \
beforelinking build build-tools buildfiles buildincludes \
- checkdpadd clean cleandepend cleandir cleanobj configure \
+ check checkdpadd clean cleandepend cleandir cleanobj configure \
depend dependall distclean distribute exe extract \
html includes install installfiles installincludes lint \
obj objlink objs objwarn realall realdepend \
- realinstall regress subdir-all subdir-depend subdir-install \
+ realinstall subdir-all subdir-depend subdir-install \
tags whereobj
.if defined(PORTNAME)
Modified: stable/10/share/mk/bsd.test.mk
==============================================================================
--- stable/10/share/mk/bsd.test.mk Thu Feb 16 04:23:45 2017 (r313789)
+++ stable/10/share/mk/bsd.test.mk Thu Feb 16 05:14:07 2017 (r313790)
@@ -68,11 +68,15 @@ _TESTS=
.include <plain.test.mk>
.include <tap.test.mk>
+# kyua automatically descends directories; only run make check on the
+# top-level directory
+.if !make(check)
.for ts in ${TESTS_SUBDIRS}
.if empty(SUBDIR:M${ts})
SUBDIR+= ${ts}
.endif
.endfor
+.endif
# it is rare for test cases to have man pages
.if !defined(MAN)
@@ -83,18 +87,13 @@ MAN=
.include <suite.test.mk>
.endif
-.if !target(realtest)
-realtest: .PHONY
+.if !target(realcheck)
+realcheck: .PHONY
@echo "$@ not defined; skipping"
.endif
-test: .PHONY
-.ORDER: beforetest realtest
-test: beforetest realtest
-
-.if target(aftertest)
-.ORDER: realtest aftertest
-test: aftertest
-.endif
+beforecheck realcheck aftercheck check: .PHONY
+.ORDER: beforecheck realcheck aftercheck
+check: beforecheck realcheck aftercheck
.include <bsd.progs.mk>
Modified: stable/10/share/mk/suite.test.mk
==============================================================================
--- stable/10/share/mk/suite.test.mk Thu Feb 16 04:23:45 2017 (r313789)
+++ stable/10/share/mk/suite.test.mk Thu Feb 16 05:14:07 2017 (r313790)
@@ -30,12 +30,19 @@ KYUAFILE?= auto
# Kyua as this is later encoded in the Kyuafile test program definitions.
#TEST_INTERFACE.<test-program>= interface-name
+# Metadata properties applicable to all test programs.
+#
+# All the variables for a test program defined in the Makefile are appended
+# to the test program's definition in the Kyuafile. This feature can be
+# used to avoid having to explicitly supply a Kyuafile in the source
+# directory, allowing the caller Makefile to rely on the KYUAFILE=auto
+# behavior defined here.
+#TEST_METADATA+= key="value"
+
# Per-test program metadata properties as a list of key/value pairs.
#
-# All the variables for a particular program are appended to the program's
-# definition in the Kyuafile. This feature can be used to avoid having to
-# explicitly supply a Kyuafile in the source directory, allowing the caller
-# Makefile to rely on the KYUAFILE=auto behavior defined here.
+# These per-test program settings _extend_ the values provided in the
+# unqualified TEST_METADATA variable.
#TEST_METADATA.<test-program>+= key="value"
.if ${KYUAFILE:tl} != "no"
@@ -43,11 +50,12 @@ FILES+= Kyuafile
FILESDIR_Kyuafile= ${TESTSDIR}
.endif
-.if ${KYUAFILE:tl} == "auto"
-CLEANFILES+= Kyuafile Kyuafile.tmp
-.endif
+.for _T in ${_TESTS}
+_TEST_METADATA.${_T}= ${TEST_METADATA} ${TEST_METADATA.${_T}}
+.endfor
.if ${KYUAFILE:tl} == "auto"
+CLEANFILES+= Kyuafile Kyuafile.tmp
Kyuafile: Makefile
@{ \
echo '-- Automatically generated by bsd.test.mk.'; \
@@ -59,10 +67,10 @@ Kyuafile: Makefile
} > ${.TARGET}.tmp
.for _T in ${_TESTS}
.if defined(.PARSEDIR)
- @echo '${TEST_INTERFACE.${_T}}_test_program{name="${_T}"${TEST_METADATA.${_T}:C/$/,/:tW:C/^/, /W:C/,$//W}}' \
+ @echo '${TEST_INTERFACE.${_T}}_test_program{name="${_T}"${_TEST_METADATA.${_T}:C/$/,/:tW:C/^/, /W:C/,$//W}}' \
>>${.TARGET}.tmp
.else
- @echo '${TEST_INTERFACE.${_T}}_test_program{name="${_T}"${TEST_METADATA.${_T}:C/^/, /:Q:S/\\ ,/,/g:S,\\,,g}}' \
+ @echo '${TEST_INTERFACE.${_T}}_test_program{name="${_T}"${_TEST_METADATA.${_T}:C/^/, /:Q:S/\\ ,/,/g:S,\\,,g}}' \
>>Kyuafile.auto.tmp
.endif
.endfor
@@ -72,9 +80,11 @@ Kyuafile: Makefile
@mv ${.TARGET}.tmp ${.TARGET}
.endif
+CHECKDIR?= ${DESTDIR}${TESTSDIR}
+
KYUA= ${LOCALBASE}/bin/kyua
-.if exists(${KYUA})
-# Definition of the "make test" target and supporting variables.
+
+# Definition of the "make check" target and supporting variables.
#
# This target, by necessity, can only work for native builds (i.e. a FreeBSD
# host building a release for the same system). The target runs Kyua, which is
@@ -83,35 +93,15 @@ KYUA= ${LOCALBASE}/bin/kyua
# Due to the dependencies of the binaries built by the source tree and how they
# are used by tests, it is highly possible for a execution of "make test" to
# report bogus results unless the new binaries are put in place.
-realtest: .PHONY
- @echo "*** WARNING: make test is experimental"
- @echo "***"
- @echo "*** Using this test does not preclude you from running the tests"
- @echo "*** installed in ${TESTSBASE}. This test run may raise false"
- @echo "*** positives and/or false negatives."
- @echo
- @${KYUA} test -k ${DESTDIR}${TESTSDIR}/Kyuafile; \
- result=0; \
- echo; \
- echo "*** Once again, note that "make test" is unsupported."; \
- test $${result} -eq 0
-.endif
-beforetest: .PHONY
-.if defined(TESTSDIR)
-.if ${TESTSDIR} == ${TESTSBASE}
-# Forbid running from ${TESTSBASE}. It can cause false positives/negatives and
-# it does not cover all the tests (e.g. it misses testing software in external).
- @echo "*** Sorry, you cannot use make test from src/tests. Install the"
- @echo "*** tests into their final location and run them from ${TESTSBASE}"
- @false
-.else
- @echo "*** Using this test does not preclude you from running the tests"
- @echo "*** installed in ${TESTSBASE}. This test run may raise false"
- @echo "*** positives and/or false negatives."
-.endif
-.else
- @echo "*** No TESTSDIR defined; nothing to do."
- @false
-.endif
- @echo
+realcheck: .PHONY
+ @if [ ! -x ${KYUA} ]; then \
+ echo; \
+ echo "kyua binary not installed at expected location (${.TARGET})"; \
+ echo; \
+ echo "Please install via pkg install, or specify the path to the kyua"; \
+ echo "package via the \$${LOCALBASE} variable, e.g. "; \
+ echo "LOCALBASE=\"${LOCALBASE}\""; \
+ false; \
+ fi
+ @${KYUA} test -k ${CHECKDIR}/Kyuafile
More information about the svn-src-stable
mailing list