Looking for speed increases in "make index" and pkg_version for ports

[Stephen Montgomery-Smith]

Jeremy Chadwick wrote:
> On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith wrote:
>>  I have been thinking a lot about looking for speed increases for "make 
>>  index" and pkg_version and things like that.  So for example, in 
>>  pkg_version, it calls "make -V PKGNAME" for every installed package. Now 
>>  "make -V PKGNAME" should be a speedy operation, but the make has to load in 
>>  and analyze bsd.port.mk, a quite complicated file with about 200,000 
>>  characters in it, when all it is needing to do is to figure out the value of 
>>  the variable PKGNAME.
> 
> I have a related question, pertaining to "make all-depends-list" and the
> utter atrocity that is the make variable ALL-DEPENDS-LIST.  If you don't
> know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in
> bsd.ports.mk.
> 
> I call it an atrocity because it's a mix of make variable expansion
> combined with sh scripting, and it's nearly impossible to read.  It's
> not commented either, so folks like myself are left thinking "What IS
> this mess?!".  It's expanded via $$(${ALL-DEPENDS-LIST}) in for loops,
> throughout several places in bsd.port.mk.
> 
> I do not entirely understand what ALL-DEPENDS-LIST is about (that should
> be apparent), but upon performing some of my benchmarks, I found this to
> be a very slow piece of bsd.port.mk.  make -V _DEPEND_DIRS is incredibly
> fast, but ALL-DEPENDS-LIST is not.
> 
> Does it need to be done this way?  Can we just iterate through all of
> the ports, call make -V _DEPEND_DIRS, then sort | uniq the results?  I
> suppose that depends on the operation (make vs. make clean vs.
> others)...
> 
> The port I used for testing some of the benchmarks was net/gacxtool.  It
> seems to be a good example of a "hefty" port.

I looked very hard at this particular piece of code.  Once you 
understand how it works, you realize that it is rather well written.  It 
basically does what you suggest, except it keeps a list of ports it has 
already checked, so that it doesn't do them over again.  This piece of 
code is as efficient as it can possibly be, given that the program has 
to recursively call make on every dependency port at least once (and as 
it happens only once).


> 
>>  I suggest rewriting "make" so that variables are only evaluated on a "need 
>>  to know" basis.  So, for example, if all we need to know is PKGNAME, there 
>>  is no need to evaluate, for example, _RUN_LIB_DEPENDS, unless the writer of 
>>  that particular port has done something like having PORTNAME depend on the 
>>  value of _RUN_LIB_DEPENDS.  So "make" should analyze all the code it is 
>>  given, and only figure it out if it is needed to do so.  This would include, 
>>  for example, figuring out .for and .if directives on a need to know basis as 
>>  well.
> 
> This sounds like a good solution.  In fact, I'm lead to believe that
> heavy reliance on /bin/sh is part of why the ports collection is slow.
> No, it's not the sole reason, but it's one of many.  I'm of the belief
> that anything we can do to migrate portions into native make would be
> benefitial.

I have done profiling tests on make, and in its current form, 
bsd.ports.mk actually spends rather little time inside of bin/sh.  Thw 
writers of bsd.ports.mk have done a very good job of minimizing the 
bin/sh calls.

> 
> That said, I'll ask this out in the open: am I the only one who sees the
> benefit of GNU make in regards to this?  There's a lot of built-in
> functions in GNU make which could help in regards to ports.  I have no
> qualms with PMake per se, but if another tool gives us what we need,
> then maybe we should consider the pros and cons of adapting that.
> There's also CMake, which is incredibly fast.
> 

Maybe I should look at the inner workings of cmake and gmake.  Maybe 
they have some good ideas.  However having looked through the source 
code of make, and also looking at the cvs logs, it does seem to be well 
written.  The only possibility I see of making it go a lot faster is a 
complete redesign, e.g. my just in time idea for processing variables.

Stephen