ports/175276: [patch] devel/py-gobject OPTIONSFILE eval order problem

Sat Feb 23 17:23:44 UTC 2013

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

1. Should this be assigned to freebsd-python@?

I realize that freebsd-gnome@ is the maintainer, but the root cause
lies with the way Python ports use PKGNAMEPREFIX, and this is not the
only affected port.

2. Allow me to elaborate on the originator's description, for those
interested in the analysis.

The common use of

PKGNAMEPREFIX=${PYTHON_PKGNAMEPREFIX}

depends on lazy evaluation, because the right-hand side is not defined
until the "pre" section of bsd.python.mk. Relatively early on in
bsd.port.mk, we get a default definition for UNIQUENAME based on
PKGNAMEPREFIX, unless LATEST_LINK is already defined, which doesn't
ordinarily happen until the "post" section of bsd.port.mk. Shortly
after that, between the "options" section and the "pre" section of
bsd.port.mk, we include bsd.options.mk which provides a default
definition of OPTIONSFILE, based on UNIQUENAME. At that point in
bsd.options.mk, we haven't yet included bsd.python.mk, so
PYTHON_PKGNAMEPREFIX is undefined. That means that when make reads the
saved options (inside the first pass through bsd.options.mk) thereby
triggering evaluation of OPTIONSFILE, it is as if we hadn't set
PKGNAMEPREFIX at all.

As the originator points out, the do-config target, where make
performs the work of writing saved options, re-evaluates OPTIONSFILE
after bsd.python.mk sets PYTHON_PKGNAMEPREFIX, because do-config is
defined in the "post" section of bsd.port.mk.

3. What ports are affected?

Any port that sets PKGNAMEPREFIX equal to a make variable that is not
defined until the "pre" section or later, and fails to work-around the
staggered evaluation by defining one of UNIQUENAME, LATEST_LINK, or
OPTIONSFILE, is broken. It turns out that Python ports are
disproportionately affected, but mainly because Python ports are heavy
users of PKGNAMEPREFIX. The other PKGNAMEPREFIXs are:

% egrep "^[A-Z_]+_PKGNAMEPREFIX" /usr/ports/Mk/* -h
APACHE_PKGNAMEPREFIX=   ap${APACHE_VERSION}-
PYTHON_PKGNAMEPREFIX?=  py*-
LUA_PKGNAMEPREFIX?=             lua${LUA_VER_STR}-
PYTHON_PKGNAMEPREFIX=   py${PYTHON_SUFFIX}-
RUBY_PKGNAMEPREFIX?=    ruby${RUBY_SUFFIX}-

But the distribution among these is heavily skewed toward Python.

% find /usr/ports -depth 3 -type f -name Makefile \
    | xargs egrep "^OPTIONS_DEFINE" -l \
    | xargs egrep "^(OPTIONSFILE|UNIQUENAME|LATEST_LINK)" -L \
    | xargs egrep '^PKGNAMEPREFIX=.*\$' -h \
    | sed -e "s/[         ]//g" \
    | sort | uniq -c | sort -n
   1 PKGNAMEPREFIX=${APACHE_PKGNAMEPREFIX}
   1 PKGNAMEPREFIX=${DMPKGNAMEPREFIX}
   1 PKGNAMEPREFIX=${DN3DPKGNAMEPREFIX}
   1 PKGNAMEPREFIX=${TGTARCH}-${TGTABI}-
   1 PKGNAMEPREFIX=php${PHP_VER}-
   2 PKGNAMEPREFIX=${LANG_PKGNAME}-
  22 PKGNAMEPREFIX=${PYTHON_PKGNAMEPREFIX}

(That's supposed to be a tab and a space in the sed command, by the way.)

So, let's focus on the 22 ports at the end.

% find /usr/ports -depth 3 -type f -name Makefile \
    | xargs egrep "^OPTIONS_DEFINE" -l \
    | xargs egrep "^(OPTIONSFILE|UNIQUENAME|LATEST_LINK)" -L \
    | xargs egrep '^PKGNAMEPREFIX=.*PYTHON' -l \
    | cut -d/ -f4-5 | sort
astro/py-RO
audio/py-karaoke
audio/py-pyaudio
databases/py-sqlkit
devel/py-bison
devel/py-gobject
devel/py-hgsubversion
dns/ldns
graphics/py-PyX
graphics/py-gdal
mail/py-spf
math/py-sympy
net/py-medusa
security/arm
security/py-volatility
security/py-yara-editor
www/py-django_compressor
www/py-imdbpy
www/py-qp
www/py-qpy
www/py-rhodecode
www/py-satchmo

I've checked every one of these by running

make config-conditional

twice in a row, and every one of them gave me a dialog the second
time, which implies that they are reading and writing saved options in
different places.

4. How can we fix this?

As I see it, there are at least the following alternatives.

A) Require each maintainer to choose and implement their preferred
work-around, defining one or more of UNIQUENAME, LATEST_LINK, or
OPTIONSFILE. This is what most affected ports do already, and what the
originator is proposing for this port. 100 ports use
PYTHON_PKGNAMEPREFIX and OPTIONS_DEFINE. 74 of those set OPTIONSFILE,
3 set LATEST_LINK, and 1 sets UNIQUENAME.

In this case we should update the documentation in bsd.python.mk and
the Porter's Handbook to make the requirement clear, and consider
implementing a validation check somewhere in /usr/ports/Mk and/or
portlint.

B) Cause part or all of the "pre" section of bsd.python.mk to be
processed earlier in bsd.port.mk, so that PYTHON_PKGNAMEPREFIX is
defined by the time we hit bsd.options.mk and need OPTIONSFILE for
reading. This would require additional analysis and testing to prevent
collateral breakage, and it would mean that bsd.python.mk becomes a
special case.

I've skimmed the portion of bsd.python.mk prior to the definition of
PYTHON_PKGNAMEPREFIX, and nothing major jumps out. If there is
interest, I would be glad to prepare a patch at which to throw darts.

C) Redefine OPTIONSFILE inside bsd.python.mk upon detecting that it
changes after defining PYTHON_PKGNAMEPREFIX, so that OPTIONSFILE is
the same when reading and writing saved options. I think we could do
this without affecting bsd.port.mk, or the ports that have already
implemented a workaround. It would mean that the default behavior when
using PYTHON_PKGNAMEPREFIX is to put saved options in
${PORT_DBDIR}/${_PNP}${PORTNAME}/options, where _PNP is any literal
used along with PYTHON_PKGNAMEPREFIX in the port's Makefile, which
some might consider a POLA violation. On the other hand, doing one
thing consistently that works is better than doing something that
breaks your port unexpectedly.

The big problem with this alternative is that PORTNAME by itself is
nowhere near unique enough to avoid conflict with other ports, and
would pretty much require bubbling up the definition of
PYTHON_PKGNAMEPREFIX from bsd.python.mk to each affected port.

5. What is the best alternative?

I find B very attractive because it frees maintainers from defining an
extra variable if they don't want (i.e. good defaults), but still
allows them to do so if they do want (i.e. good mechanism). It may be
difficult, hackish, and error-prone though.

Option A would be easiest, and least traumatic both to individual
ports and to the ports infrastructure itself. For this reason alone, A
is probably the right choice for now.

Sadly, C may be a complete non-starter due to the uniqueness problem,
but I wouldn't rule it out completely as a long-term follow-up to A.
The way I see it working out is in three phases. Phase one is to
implement option A but also invite maintainers to replace

PKGNAMEPREFIX= ${PYTHON_PKGNAMEPREFIX}

with

PKGNAMEPREFIX= py${PYTHON_SUFFIX}-

Phase two is to implement option C. Phase three is to invite
maintainers to remove the option A work-around if they like the
then-default behavior.

6. Conclusion.

I invite commentary and criticism, especially on the potential
resolutions I proposed. When we reach consensus, I will set about
preparing some patches, if need be, and seeking the help of a friendly
committer.

Thank you for your kind indulgence.

Cheers,
John
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQEcBAEBAgAGBQJRKPsXAAoJEEdKvTwaez9w6yEIALFz+xrYLMdR1AhcPE2jEBd6
uR4dOZye8PQFTHbvhA/t20NFTroalr2kXF49+PTqR6kCFes+vNgjIlWUdKsIngYk
y5x32f60Bd/TtqPo6M2aeOE/M322U6cIH5jJhh3EBTEpm+Upd9enIetxR0NpjTnP
G+6yf8e7P4oBaYGSk01i3pah00OR2YeC87rtcEdgs1sM94PjxbXZGcuA+K9UbgVQ
2WB8Z4IvrD3d2UqRnC8TRq1/bZyiPSHKNeMFBRJZ4gFe/wr9G0txDnH1LTy/q0Gq
kVHvdbApLYytMX/VmMMgDRnbzGS/kDMvIED8dJnwWf9pMLmzsi0pcVX/vH0m1Vw=
=q6eG
-----END PGP SIGNATURE-----