Re: git: c2afef5c2587 - main - Features/testing.mk: Introduce the TESTING_UNSAFE knob.

On 2024-03-02T08:45:20.000+01:00, Gleb Popov <arrowd@freebsd.org> wrote:
>  On Fri, Mar 1, 2024 at 11:20 PM Daniel Engberg
> <daniel.engberg.lists@pyret.net> wrote:
> 
> >   
> >  Hi,
> > 
> >  Thanks for adding this, however this won't work on ports that uses TEST menu option as far as I can tell which is a relatively common occurence?
> > 
> >  Se also https://cgit.freebsd.org/ports/tree/Mk/bsd.options.desc.mk#n495
> >  Best regards,
> >  Daniel
> > 
>  
> There is a subtle overlap between Features and OPTIONS that I still
> have not fully grasped.
> 
> A Feature can be thought of as an OPTION that is generic enough to be
> applied to every port. For instance, almost every software can be
> built in the "debug" mode (whatever that means), which makes DEBUG a
> good candidate for a Feature and not an OPTION. However, some software
> may have additional toggles that are not hooked into the standard way
> of enabling debug mode for a given build system.
> 
> A CMake example - there is a standard CMAKE_BUILD_TYPE variable, but
> the software may also define sort of ENABLE_EXTRA_CHECKS option, which
> further improve debugging experience but is not enabled by
> CMAKE_BUILD_TYPE.
> 
> In such cases it might have a sense to define the DEBUG option and
> then do DEBUG_CMAKE_ON= -DENABLE_EXTRA_CHECKS=yes in the port.
> 
> In case of LTO a software may require some additional dependency, so
> that just setting WITH_LTO=yes would not be enough. So this is another
> case where the option-corresponding-to-a-feature is required.
> 
> Note that it is still possible to build WITH_DEBUG=yes but
> WITHOUT=DEBUG. It will enable debugging in general, but not enable
> some specific debugging features for the given port. This hints that
> Features and OPTIONS are separate ideas and probably should coexist.
> 
> Getting back to the TESTING feature, I agree that the TEST option
> doesn't really make sense as it is. At least we should change its
> description and remove the "and/or run" part, because running tests is
> now (and should be) a Feature-level decision.
> Maybe the TEST option doesn't make sense at all now, but like I said I
didn't get my head over it yet. Let's see how it goes.

In my experience and looking at other ports the framework lacks functionality and/or causes execution to fail which is why the menu option is utilized. Just looking at cmake:testing we're already working around the stock TEST_* options and functionality available.

Best regards,
Daniel