Looking for speed increases in "make index" and pkg_version for ports

[Hartmut Brandt]

Matthew Seaman wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> Stephen Montgomery-Smith wrote:
>> I have been thinking a lot about looking for speed increases for "make
>> index" and pkg_version and things like that.  So for example, in
>> pkg_version, it calls "make -V PKGNAME" for every installed package. Now
>> "make -V PKGNAME" should be a speedy operation, but the make has to load
>> in and analyze bsd.port.mk, a quite complicated file with about 200,000
>> characters in it, when all it is needing to do is to figure out the
>> value of the variable PKGNAME.
> 
> pkg_version is one thing -- but to build the INDEX you need to extract
> at least the values of the following variables:
> 
>       PKGNAME
>       .CURDIR
>       PREFIX
>       COMMENT
>       DESCR
>       MAINTAINER
>       CATEGORIES
>       EXTRACT_DEPENDS
>       PATCH_DEPENDS
>       FETCH_DEPENDS
>       BUILD_DEPENDS
>       RUN_DEPENDS
>       LIB_DEPENDS
> 
> Plus you need to grep in the referenced pkg-descr file for any WWW
> links.  I also extract the values of:
> 
>       MASTER_PORT
>       .MAKEFILE_LIST
>       SUBDIR
> 
> for my FreeBSD::Portindex stuff.
> 
> Trouble is, by the time you've extracted all that lot, you have pretty
> much done the same level of variable processing as you would were you
> actually going to build the port.
> 
> One thing that would speed up this process would be a make option
> to just do parsing of the Makefile and variable expansion, without
> calling stat(2) on all the various sources and dependencies involved.
> 
> For instance:
> 
> happy-idiot-talk:...ports/databases/mysql-connector-java:% truss make -V PKGNAME |& grep stat | wc -l
>       49
> 
> It is quite instructive to see what files make(1) touches while doing
> that.  At least half of them are irrelevant if all make(1) is going to
> do is print out the values of some variables.  Multiply that by 17,000
> and it adds up to a big waste of effort.

Are you sure? A good deal of the stats() is make poking around for the 
make infrastructure. This should be in the cache. And there are a couple 
of stats for the *done* files that might be avoided by doing something 
in the ports infrastructure. But as I already said in my previous mail: 
numbers, please, no guessing.

harti