Looking for speed increases in "make index"

[Michel Talon]

Stephen Montgomery-Smith said:

> I suggest rewriting "make" so that variables are only evaluated on a 
> "need to know" basis. ....
> or "I have tried to do this."

Of course a lot of people have thinked about it, and quickly realized
that it was not going to work. In the bsd.ports.mk, evaluation of one
variable may be dependent on some conditional, which may itself be
dependant on some other variable, appearing as some target. This
constantly happens in bsd.ports.mk.
 
If you think about that, you convince yourself that a "reduced make"
needs to understand targets, needs to understand conditionals, and needs
to evaluate not only the variable under consideration, but but possibly
any other. In other words, you need a full blown make.

To gain some performance, a first idea would be to simplify
bsd.ports.mk. I am convinced that a substantial part of the 4000 lines
are historical crap which serve no useful purpose. There are tons of
variables who have probably purely anecdotical interest. There are
targets which could be happily suppressed. Of course, due to the
complexity of bsd.ports.mk, rewriting it is certainly not an easy task.
There is a freebsd port whose aim is to rewrite it, i don't know how
far they are. The NetBSD people have completely rewritten the system, i
have no idea if the new one is faster. The OpenBSD people have also
rewritten bsd.ports.mk, with apparently much faster makefile.

A second idea would be to multithread make, since modern machines will
have a lot of cores. At present make -j <something> forks submakes
and waits for their completion. This doesn't help for the problem at
hand. However, multithreading the execution of a single makefile is
certainly not trivial due to the interdependencies.

Anyways, one of the things which cannot be a big factor is reading
bsd.ports.mk from hard disk. It is certainly cached in memory when you
run make index. On the other hand it is so big that it probably doesn't
fit in cache, or probably only fits in caches of machines generously
endowed. I have remarked that the difference of execution speed of make
index between machines of similar speed but very different cache size
(i am thinking Pentium 4 versus Core 2 Duo) is striking. It is less
than 10 minutes on the big cache machine versus 30 minutes on the small
cache one. If the cache effect is indeed dominant, then reducing the
size of bsd.ports.mk by all possible means would be very beneficial.

By the way an alternative system would be to use something other than
make to do the job. This is what the Gentoo people are doing, using
first a python based system, and now a C++ one (paludis). Here also i
don't have any idea if it is faster.




-- 

Michel TALON