Looking for speed increases in "make index" and pkg_version for ports

[Peter Jeremy]

On 2007-May-27 15:52:16 -0500, Stephen Montgomery-Smith <stephen at math.missouri.edu> wrote:
> I have been thinking a lot about looking for speed increases for "make 
> index" and pkg_version and things like that.  So for example, in 
> pkg_version, it calls "make -V PKGNAME" for every installed package. Now 
> "make -V PKGNAME" should be a speedy operation, but the make has to load in 
> and analyze bsd.port.mk, a quite complicated file with about 200,000 
> characters in it, when all it is needing to do is to figure out the value of 
> the variable PKGNAME.

This would be trivial if some packages didn't change names depending
on options and what was installed.  I agree that parsing a 210KB file
17,000 times is not going to be fast.  Especially since some ports
include bsd.ports.mk multiple times...

> I suggest rewriting "make" so that variables are only evaluated on a "need 
> to know" basis.

This sounds like a good idea but I suspect it's not going to be
feasible.  The biggest problem I see is that the make language allows
variables to be expanded either when they are assigned or when they
are referenced.  If a variable expansion is delayed from the
assignment to the first use then the expansion must be performed using
the state of make as it was when the variable was assigned.  The cost
of keeping this state probably exceeds the cost of actually evaluating
the variable.

>  So, for example, if all we need to know is PKGNAME, there 
> is no need to evaluate, for example, _RUN_LIB_DEPENDS, unless the writer of 
> that particular port has done something like having PORTNAME depend on the 
> value of _RUN_LIB_DEPENDS.

Rather than trying to develop a tool that can quickly expand PKGNAME
irrespective of what convoluted code the author has used, how about a
partial solution?  For most ports, PKGNAME depends solely on 3 or 4
variables that are statically defined in the port Makefile.  The
obvious solution would seem to be to develop a script that can handle
the easy cases itself and punt the difficult cases back to make.  The
definition of 'easy' can be adjusted over time to increase performance.
This approach would seem to have a relatively low bar to entry whilst
offering good effort/performance tradeoff at the low end.

The various depends lists would seem amenable to the same approach -
though the entry level tool will have far lower coverage due to the
extensive use of USE_GNOME=... and similar 'macro'-style constructs.

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-ports/attachments/20070529/2b2662d7/attachment.pgp