ports/175276: [patch] devel/py-gobject OPTIONSFILE eval order problem

Tue Mar 19 18:00:02 UTC 2013

The following reply was made to PR ports/175276; it has been noted by GNATS.

From: Koop Mast <kwm at rainbow-runner.nl>
To: Jeremy Messenger <mezz.freebsd at gmail.com>
Cc: "John W. O'Brien" <john at saltant.com>, gnome at freebsd.org, 
 FreeBSD Ports Mailing List <ports at freebsd.org>,
 Baptiste Daroussin <bapt at freebsd.org>, bug-followup at FreeBSD.org
Subject: Re: ports/175276: [patch] devel/py-gobject OPTIONSFILE eval order
 problem
Date: Tue, 19 Mar 2013 18:50:08 +0100

 On 19-3-2013 17:56, Jeremy Messenger wrote:
 > Sorry, this is way too long to read. I will just skip the read and
 > post my suggest of solution to this problem in the top of your email.
 > I think the OPTIONS needs to change from ${UNIQUENAME} to
 > ${PKGORIGIN:S/\//_/}. It will be looked like
 > "${PORT_DBDIR}/cat_port/options". Here's example:
 >
 > In bsd.options.mk:
 > -----------------------------------
 > [...]
 >
 > OPTIONSFILE?=	${PORT_DBDIR}/${PKGORIGIN:S/\//_/}/options
 > -----------------------------------
 >
 > Then add compatible in somewhere like this:
 > -----------------------------------
 > .if exist (${PORT_DBDIR}/${UNIQUENAME}/options)
 > 	@${MV} ${PORT_DBDIR}/${UNIQUENAME} ${PORT_DBDIR}/${PKGORIGIN:S/\//_/}
 > .endif
 > -----------------------------------
 >
 > Then teach the portmaster about if the port has been moved to the
 > different category or renamed (by read MOVED) then change the
 > ${PORT_DBDIR}/${PKGORIGIN:S/\//_/}.
 >
 > What do anyone think of my suggest solution? I haven't test anything
 > at all, which it's just what I have in my mind right now.

 I was thinking of just axing the option completely and moving libffi to 
 lib_depend. glib20 already depends on libffi, so we also could get away 
 with removing it completely. But that doesn't resolve that this problem 
 might appear in other ports.

 -Koop

 >
 > On Sat, Feb 23, 2013 at 11:30 AM, John W. O'Brien <john at saltant.com> wrote:
 >> The following reply was made to PR ports/175276; it has been noted by GNATS.
 >>
 >> From: "John W. O'Brien" <john at saltant.com>
 >> To: bug-followup at FreeBSD.org, jhein at symmetricom.com
 >> Cc: freebsd-python at freebsd.org
 >> Subject: Re: ports/175276: [patch] devel/py-gobject OPTIONSFILE eval order
 >>   problem
 >> Date: Sat, 23 Feb 2013 12:23:35 -0500
 >>
 >>   -----BEGIN PGP SIGNED MESSAGE-----
 >>   Hash: SHA1
 >>
 >>   1. Should this be assigned to freebsd-python@?
 >>
 >>   I realize that freebsd-gnome@ is the maintainer, but the root cause
 >>   lies with the way Python ports use PKGNAMEPREFIX, and this is not the
 >>   only affected port.
 >>
 >>
 >>   2. Allow me to elaborate on the originator's description, for those
 >>   interested in the analysis.
 >>
 >>   The common use of
 >>
 >>   PKGNAMEPREFIX=${PYTHON_PKGNAMEPREFIX}
 >>
 >>   depends on lazy evaluation, because the right-hand side is not defined
 >>   until the "pre" section of bsd.python.mk. Relatively early on in
 >>   bsd.port.mk, we get a default definition for UNIQUENAME based on
 >>   PKGNAMEPREFIX, unless LATEST_LINK is already defined, which doesn't
 >>   ordinarily happen until the "post" section of bsd.port.mk. Shortly
 >>   after that, between the "options" section and the "pre" section of
 >>   bsd.port.mk, we include bsd.options.mk which provides a default
 >>   definition of OPTIONSFILE, based on UNIQUENAME. At that point in
 >>   bsd.options.mk, we haven't yet included bsd.python.mk, so
 >>   PYTHON_PKGNAMEPREFIX is undefined. That means that when make reads the
 >>   saved options (inside the first pass through bsd.options.mk) thereby
 >>   triggering evaluation of OPTIONSFILE, it is as if we hadn't set
 >>   PKGNAMEPREFIX at all.
 >>
 >>   As the originator points out, the do-config target, where make
 >>   performs the work of writing saved options, re-evaluates OPTIONSFILE
 >>   after bsd.python.mk sets PYTHON_PKGNAMEPREFIX, because do-config is
 >>   defined in the "post" section of bsd.port.mk.
 >>
 >>
 >>   3. What ports are affected?
 >>
 >>   Any port that sets PKGNAMEPREFIX equal to a make variable that is not
 >>   defined until the "pre" section or later, and fails to work-around the
 >>   staggered evaluation by defining one of UNIQUENAME, LATEST_LINK, or
 >>   OPTIONSFILE, is broken. It turns out that Python ports are
 >>   disproportionately affected, but mainly because Python ports are heavy
 >>   users of PKGNAMEPREFIX. The other PKGNAMEPREFIXs are:
 >>
 >>   % egrep "^[A-Z_]+_PKGNAMEPREFIX" /usr/ports/Mk/* -h
 >>   APACHE_PKGNAMEPREFIX=   ap${APACHE_VERSION}-
 >>   PYTHON_PKGNAMEPREFIX?=  py*-
 >>   LUA_PKGNAMEPREFIX?=             lua${LUA_VER_STR}-
 >>   PYTHON_PKGNAMEPREFIX=   py${PYTHON_SUFFIX}-
 >>   RUBY_PKGNAMEPREFIX?=    ruby${RUBY_SUFFIX}-
 >>
 >>   But the distribution among these is heavily skewed toward Python.
 >>
 >>   % find /usr/ports -depth 3 -type f -name Makefile \
 >>       | xargs egrep "^OPTIONS_DEFINE" -l \
 >>       | xargs egrep "^(OPTIONSFILE|UNIQUENAME|LATEST_LINK)" -L \
 >>       | xargs egrep '^PKGNAMEPREFIX=.*\$' -h \
 >>       | sed -e "s/[         ]//g" \
 >>       | sort | uniq -c | sort -n
 >>      1 PKGNAMEPREFIX=${APACHE_PKGNAMEPREFIX}
 >>      1 PKGNAMEPREFIX=${DMPKGNAMEPREFIX}
 >>      1 PKGNAMEPREFIX=${DN3DPKGNAMEPREFIX}
 >>      1 PKGNAMEPREFIX=${TGTARCH}-${TGTABI}-
 >>      1 PKGNAMEPREFIX=php${PHP_VER}-
 >>      2 PKGNAMEPREFIX=${LANG_PKGNAME}-
 >>     22 PKGNAMEPREFIX=${PYTHON_PKGNAMEPREFIX}
 >>
 >>   (That's supposed to be a tab and a space in the sed command, by the way.)
 >>
 >>   So, let's focus on the 22 ports at the end.
 >>
 >>   % find /usr/ports -depth 3 -type f -name Makefile \
 >>       | xargs egrep "^OPTIONS_DEFINE" -l \
 >>       | xargs egrep "^(OPTIONSFILE|UNIQUENAME|LATEST_LINK)" -L \
 >>       | xargs egrep '^PKGNAMEPREFIX=.*PYTHON' -l \
 >>       | cut -d/ -f4-5 | sort
 >>   astro/py-RO
 >>   audio/py-karaoke
 >>   audio/py-pyaudio
 >>   databases/py-sqlkit
 >>   devel/py-bison
 >>   devel/py-gobject
 >>   devel/py-hgsubversion
 >>   dns/ldns
 >>   graphics/py-PyX
 >>   graphics/py-gdal
 >>   mail/py-spf
 >>   math/py-sympy
 >>   net/py-medusa
 >>   security/arm
 >>   security/py-volatility
 >>   security/py-yara-editor
 >>   www/py-django_compressor
 >>   www/py-imdbpy
 >>   www/py-qp
 >>   www/py-qpy
 >>   www/py-rhodecode
 >>   www/py-satchmo
 >>
 >>   I've checked every one of these by running
 >>
 >>   make config-conditional
 >>
 >>   twice in a row, and every one of them gave me a dialog the second
 >>   time, which implies that they are reading and writing saved options in
 >>   different places.
 >>
 >>
 >>   4. How can we fix this?
 >>
 >>   As I see it, there are at least the following alternatives.
 >>
 >>   A) Require each maintainer to choose and implement their preferred
 >>   work-around, defining one or more of UNIQUENAME, LATEST_LINK, or
 >>   OPTIONSFILE. This is what most affected ports do already, and what the
 >>   originator is proposing for this port. 100 ports use
 >>   PYTHON_PKGNAMEPREFIX and OPTIONS_DEFINE. 74 of those set OPTIONSFILE,
 >>   3 set LATEST_LINK, and 1 sets UNIQUENAME.
 >>
 >>   In this case we should update the documentation in bsd.python.mk and
 >>   the Porter's Handbook to make the requirement clear, and consider
 >>   implementing a validation check somewhere in /usr/ports/Mk and/or
 >>   portlint.
 >>
 >>   B) Cause part or all of the "pre" section of bsd.python.mk to be
 >>   processed earlier in bsd.port.mk, so that PYTHON_PKGNAMEPREFIX is
 >>   defined by the time we hit bsd.options.mk and need OPTIONSFILE for
 >>   reading. This would require additional analysis and testing to prevent
 >>   collateral breakage, and it would mean that bsd.python.mk becomes a
 >>   special case.
 >>
 >>   I've skimmed the portion of bsd.python.mk prior to the definition of
 >>   PYTHON_PKGNAMEPREFIX, and nothing major jumps out. If there is
 >>   interest, I would be glad to prepare a patch at which to throw darts.
 >>
 >>   C) Redefine OPTIONSFILE inside bsd.python.mk upon detecting that it
 >>   changes after defining PYTHON_PKGNAMEPREFIX, so that OPTIONSFILE is
 >>   the same when reading and writing saved options. I think we could do
 >>   this without affecting bsd.port.mk, or the ports that have already
 >>   implemented a workaround. It would mean that the default behavior when
 >>   using PYTHON_PKGNAMEPREFIX is to put saved options in
 >>   ${PORT_DBDIR}/${_PNP}${PORTNAME}/options, where _PNP is any literal
 >>   used along with PYTHON_PKGNAMEPREFIX in the port's Makefile, which
 >>   some might consider a POLA violation. On the other hand, doing one
 >>   thing consistently that works is better than doing something that
 >>   breaks your port unexpectedly.
 >>
 >>   The big problem with this alternative is that PORTNAME by itself is
 >>   nowhere near unique enough to avoid conflict with other ports, and
 >>   would pretty much require bubbling up the definition of
 >>   PYTHON_PKGNAMEPREFIX from bsd.python.mk to each affected port.
 >>
 >>
 >>   5. What is the best alternative?
 >>
 >>   I find B very attractive because it frees maintainers from defining an
 >>   extra variable if they don't want (i.e. good defaults), but still
 >>   allows them to do so if they do want (i.e. good mechanism). It may be
 >>   difficult, hackish, and error-prone though.
 >>
 >>   Option A would be easiest, and least traumatic both to individual
 >>   ports and to the ports infrastructure itself. For this reason alone, A
 >>   is probably the right choice for now.
 >>
 >>   Sadly, C may be a complete non-starter due to the uniqueness problem,
 >>   but I wouldn't rule it out completely as a long-term follow-up to A.
 >>   The way I see it working out is in three phases. Phase one is to
 >>   implement option A but also invite maintainers to replace
 >>
 >>   PKGNAMEPREFIX= ${PYTHON_PKGNAMEPREFIX}
 >>
 >>   with
 >>
 >>   PKGNAMEPREFIX= py${PYTHON_SUFFIX}-
 >>
 >>   Phase two is to implement option C. Phase three is to invite
 >>   maintainers to remove the option A work-around if they like the
 >>   then-default behavior.
 >>
 >>
 >>   6. Conclusion.
 >>
 >>   I invite commentary and criticism, especially on the potential
 >>   resolutions I proposed. When we reach consensus, I will set about
 >>   preparing some patches, if need be, and seeking the help of a friendly
 >>   committer.
 >>
 >>   Thank you for your kind indulgence.
 >>
 >>   Cheers,
 >>   John
 >>   -----BEGIN PGP SIGNATURE-----
 >>   Version: GnuPG v1.4.11 (GNU/Linux)
 >>   Comment: Using GnuPG with undefined - http://www.enigmail.net/
 >>
 >>   iQEcBAEBAgAGBQJRKPsXAAoJEEdKvTwaez9w6yEIALFz+xrYLMdR1AhcPE2jEBd6
 >>   uR4dOZye8PQFTHbvhA/t20NFTroalr2kXF49+PTqR6kCFes+vNgjIlWUdKsIngYk
 >>   y5x32f60Bd/TtqPo6M2aeOE/M322U6cIH5jJhh3EBTEpm+Upd9enIetxR0NpjTnP
 >>   G+6yf8e7P4oBaYGSk01i3pah00OR2YeC87rtcEdgs1sM94PjxbXZGcuA+K9UbgVQ
 >>   2WB8Z4IvrD3d2UqRnC8TRq1/bZyiPSHKNeMFBRJZ4gFe/wr9G0txDnH1LTy/q0Gq
 >>   kVHvdbApLYytMX/VmMMgDRnbzGS/kDMvIED8dJnwWf9pMLmzsi0pcVX/vH0m1Vw=
 >>   =q6eG
 >>   -----END PGP SIGNATURE-----
 >> _______________________________________________
 >> freebsd-gnome at freebsd.org mailing list
 >> http://lists.freebsd.org/mailman/listinfo/freebsd-gnome
 >> To unsubscribe, send any mail to "freebsd-gnome-unsubscribe at freebsd.org"
 >
 >