Looking for speed increases in "make index"

[Stephen Montgomery-Smith]

On Mon, 28 May 2007, Michel Talon wrote:

> Stephen Montgomery-Smith said:
>
>> I suggest rewriting "make" so that variables are only evaluated on a
>> "need to know" basis. ....
>> or "I have tried to do this."
>
> Of course a lot of people have thinked about it, and quickly realized
> that it was not going to work. In the bsd.ports.mk, evaluation of one
> variable may be dependent on some conditional, which may itself be
> dependant on some other variable, appearing as some target. This
> constantly happens in bsd.ports.mk.
>
> If you think about that, you convince yourself that a "reduced make"
> needs to understand targets, needs to understand conditionals, and needs
> to evaluate not only the variable under consideration, but but possibly
> any other. In other words, you need a full blown make.

Actually I have thought quite a lot about this over the last couple of 
weeks.  You are correct in that the "make" really does have to be a full 
blown make.  What I am suggesting is something rather different - a rather 
sophisticated make that works hard to only that which it really has to do. 
And it certainly would require a lot of sophistication, precisely for the 
reasons you state.

>
> To gain some performance, a first idea would be to simplify
> bsd.ports.mk. I am convinced that a substantial part of the 4000 lines
> are historical crap which serve no useful purpose. There are tons of
> variables who have probably purely anecdotical interest. There are
> targets which could be happily suppressed. Of course, due to the
> complexity of bsd.ports.mk, rewriting it is certainly not an easy task.
> There is a freebsd port whose aim is to rewrite it, i don't know how
> far they are. The NetBSD people have completely rewritten the system, i
> have no idea if the new one is faster. The OpenBSD people have also
> rewritten bsd.ports.mk, with apparently much faster makefile.
>

I rewrote a bsd.ports.mk in which ALL targets were removed, and all it did 
was evalute variables.  (I wrote a perl script that did this 
automatically.)  The net effect was no speed gain.  In other words, 
processing unnecessary targets is not the problem.

I really have looked at bsd.ports.mk a lot, and given that it is written 
in the context of what make requires, it seems to be rather well and 
efficiently written.  I did find one speed increase in bsd.gnome.mk, which 
I am kind of proud of, but even that only gained about a 5-10% speed 
increase in "make index."


> A second idea would be to multithread make, since modern machines will
> have a lot of cores. At present make -j <something> forks submakes
> and waits for their completion. This doesn't help for the problem at
> hand. However, multithreading the execution of a single makefile is
> certainly not trivial due to the interdependencies.
>

I don't think multithreaded make will help in this situation.

> Anyways, one of the things which cannot be a big factor is reading
> bsd.ports.mk from hard disk. It is certainly cached in memory when you
> run make index. On the other hand it is so big that it probably doesn't
> fit in cache, or probably only fits in caches of machines generously
> endowed. I have remarked that the difference of execution speed of make
> index between machines of similar speed but very different cache size
> (i am thinking Pentium 4 versus Core 2 Duo) is striking. It is less
> than 10 minutes on the big cache machine versus 30 minutes on the small
> cache one. If the cache effect is indeed dominant, then reducing the
> size of bsd.ports.mk by all possible means would be very beneficial.
>

My suspicion is that the timne taken to read bsd.ports.mk from the hard 
disk is a rather small part of what takes it so long.  Compare doing a 
"make" with a simple "cat" and the time differences are very great.  I 
think make spends far more time processing the file than reading it.

> By the way an alternative system would be to use something other than
> make to do the job. This is what the Gentoo people are doing, using
> first a python based system, and now a C++ one (paludis). Here also i
> don't have any idea if it is faster.
>

It would be nice to find a solution that doesn't involve redesigning the 
whole ports process from scratch.


>
>
>
> -- 
>
> Michel TALON
>
> _______________________________________________
> freebsd-hackers at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org"
>