make WITH[OUT]_* more useful?

[Warner Losh]

Hi Simon,

thanks for the good dialog, and the patient explanation of things. Sorry I’ve taken
a bit to reply...

On Apr 1, 2014, at 1:52 PM, Simon J. Gerraty <sjg at juniper.net> wrote:

> On Tue, 1 Apr 2014 12:57:57 -0600, Warner Losh writes:
>>> .if !defined(WITHOUT_CTF) && !defined(NO_CTF)
>>> WITH_CTF=3Dyes
>>> .endif
>>> =20
>>> else you get errors during build world.
>> 
>> That's a bug I plan on fixing. 
> 
> Agreed, and cool.

I almost have this bug fixed. There’s one instance where it isn’t that helps
illustrate the point you are making.

>>> That's  not so much a problem for existing options, but makes it
>>> impractical to leverage the mechanism for things you might want to set
>>> during sys.mk - like whether to use meta mode or auto objdir creation.
>> 
>> I=92ll have to think about this point. It is a good point, but I=92m =
>> unsure how
>> the proposed changes would fix that. Perhaps you could explain that a =
>> bit.
> 
> Sure.  For the sake of argument assume I can put something like this in
> local.sys.mk:
> 
> OPTIONS_DEFAULT_NO+= \
> 	META_MODE
> 
> OPTIONS_DEFAULT_YES+= \
> 	AUTO_OBJ
> 
> .include "options.mk"
> 
> .if ${MK_AUTO_OBJ} == "yes"
> .include "auto.obj.mk"
> .endif
> .if ${MK_META_MODE} == "yes"
> .include "meta.sys.mk"
> .endif
> 
> and both
> 
> MK_META_MODE={yes,no}
> MK_AUTO_OBJ={yes,no}
> 
> which are needed during sys.mk can be influenced by user's -DWITH[OUT_*

OK. A bit of a contrived example, but I suppose if I understood the meta mode
a bit better I might think differently :)

I’m a bit hesitant, though, to have this affect sys.mk because that affects all users
of make, not just /usr/src.

>> Setting defaults in make.conf, and then overriding them is tough because
>> bmake doesn't have a tracking system to know where in the food-chain
>> different variables are set. However, this is also a good point, but I
>> must be being dense to see how your proposal would fix this.
> 
> It "fixes" it by not throwing an error when both WITH_FOO and WITHOUT_FOO
> are defined.  Instead, you decree that one wins (eg WITHOUT_FOO
> overrides WITH_FOO).

In some cases, you can declare a winner. In other cases that might be harder
to do. With almost all options defaulting to YES, having WITHOUT win makes
good sense. When there’s more of a mix, I’m less sure.

> You could also address the conflict by giving preference to command line.
> ${.MAKEFLAGS:MWITH*} will give you any WITH_* and WITHOUT_* from command
> line.  
> 
> There might be a rare case where an error is desired on conflict, but
> if you can sensibly resolve it that is better.
> 
>>> I have a number of knobs to be set during sys.mk
>>> AUTO_OBJ automatically create objdirs
>>> META_MODE use meta mode
>> 
>> objdirs set in sys.mk? 
> 
> yes.
> 
>> I thought that was done bad.obj.mk since it isn't part of the global system.
> 
> Traditionally done in bsd.obj.mk - but that requires a separate
> invocation of make.

Right, but can’t that be done automatically w/o that extra invocation?

> The junos build for example has auto-created objdirs for 10+ years.
> The projects/bmake build does the same.
> It is a good way to ensure consistent/reliable results and avoid
> probelms when users forget to 'make obj'.
> 
> When you have to support 2000+ developers you want to be able to
> make it hard for them to shoot themselves in the foot.
> 
> If making objdirs automatically, you really want to do it early, since
> the result influences how make then behaves.

True, all good stuff. Just got caught up quibbling over its need to be
in sys.mk vs enhancing bsd.obj.mk to do this...

>> META_MODE might belong there.
>> 
>>> setting MK_* is fine, but it is nice if you allow the user to interact
>>> using WITH[OUT]_*, and for that it is best if sys.mk can safely =
>> include
>>> something like options.mk
>> 
>> Not sure how creating a new file solves the bsd.own.mk problem, apart
>> perhaps from some name space pollution differences.
> 
> An options.mk which does nothing except process the list of
> options provided to it, is always safe to include even by sys.mk.
> Whereas bsd.own.mk does other stuff which makes it unsuitable to include 
> eg. during the early stages of buildworld.
> bsd.own.mk can of course set its own list of options and then include
> the same options.mk a simple matter of refactoring to allow re-use.

Back to the NO_CTF stuff I talked about above: I totally get this. If you look at
bsd.own.mk it is in 3 sections (plus a stray line at the end that really belongs in
the first section.

# section 1 - stuff about ownership
# section 2 - stuff to do options
# section 3 - setting defaults based on options

It is this last section that’s a problem to CTF specifically. Normally, we have
a paradigm in the tree where we do

.include <bsd.own.mk>

.if ${MK_FOO} != “no”
CFLAGS+=-DSUPPORT_FOO
.endif

There’s one place in the tree that wants to turn off CTF. This is mostly fixed by
just setting MK_CTF=no in that makefile after we include bsd.own.mk. I say
mostly fixed because we wind up with a NULL command where we really want
a @: command, though the former I believe is harmless but verbose. Except
one could unset WITH_CTF (which doesn’t completely work, it still shows
up as defined) and set WITHOUT_CTF before bsd.own.mk and it would work,
modulo this bug.

This can really only be fixed by making bsd.own.mk look more like

# section 1 -same
.include <bsd.options.mk>
# section 3 - same

with bsd.options.mk looking a lot like section 2 of bsd.own.mk.  Also, we’d have
to include bsd.options.mk at the head of the Makefile rather than bsd.own.mk
where we do now.

This sounds a lot like what you’re trying to describe, though placement of
bsd.options.mk may be different than I described. The only reason we
need it where I suggested it is compatibility with the past. Though we may
be able to get away with it in sys.mk, I’m hesitant to place it in there because
that’s global to everything, including ports, etc. Plus, I think it is too early, due
to meta MK_ variables, that I talk about below.

>> Yea, back to the point I don=92t understand: how is your new file going =
>> to be
>> materially different than the current mechanism. I=92m OK with change, =
>> but I
>> need to understand how the change is going to make things better.
> 
> Separating the mechanism of option processing from the definition of
> options makes re-use easier - see above, sorry I don't know how to make
> it clearer.
> 
> The other aspects you have at least partially addressed already, by
> allowing MK_* to be set independently.
> Though:
> 
> .if defined(MK_${var})
> .if ${.MAKE.LEVEL} == 0
> .error MK_${var} can't be set by a user.
> .endif
> 
> would seem to negate that.  Why can a makefile at level 0 not set MK_*?

Well, the notion now is that if you want to test MK_* variables, you need to
include bsd.own.mk first. The notion I was going for with the above is that you’d
have to include bsd.own.mk first, then set the MK_* variables in your Makefile,
so the tests won’t be hit.

But there’s a problem even if we take the approach above. Section 2 in bsd.own.mk
is actually two separate sections. One that sets the MK_* variables based on
WITH_ or WITHOUT_ and then a second section that cascades the MK_ variables
to other MK_ variables (like MK_CRYPT==no turning of OpenSSL, Kerberos, etc). If
you wanted to set one of those variables in your Makefile, you’d have a chicken
and egg problem. If you set it before bsd.options.mk, then you’d get the cascade effects
but hit the warning. If you set it after, you dodge the warning, but don’t get the cascade.

This problem suggests, perhaps, that the test be deleted.

> The outstanding question is dealing with conflict when both WITH_FOO and
> WITHOUT_FOO are defined.

True. That’s a tougher problem than you might think on first blush, as we’ve
been talking about. For now, I’d suggest WITHOUT wins, and we see how far
that gets us and what problems come up. Anything else seems like it would
get uber complicated quite quickly as we play whack-a-mole with different edge
use cases.

Warner