Looking for speed increases in "make index"

[Peter Jeremy]

On 2007-May-28 00:15:28 +0200, Michel Talon <talon at lpthe.jussieu.fr> wrote:
>Of course a lot of people have thinked about it, and quickly realized
>that it was not going to work. In the bsd.ports.mk, evaluation of one
>variable may be dependent on some conditional, which may itself be
>dependant on some other variable, appearing as some target. This
>constantly happens in bsd.ports.mk.
...
>To gain some performance, a first idea would be to simplify
>bsd.ports.mk. I am convinced that a substantial part of the 4000 lines
>are historical crap which serve no useful purpose. There are tons of
>variables who have probably purely anecdotical interest. There are
>targets which could be happily suppressed.

These two paragraphs are contradictory.  If bsd.ports.mk really is
4000 lines of historical crap and full full of unused variables than
an "evaluate on demand" approach would be a big win.

>A second idea would be to multithread make, since modern machines will
>have a lot of cores.

Not yet.  Dual-core machines are still relatively new.  Of the seven
hosts that I regularly use, only one is dual core.  Threading adds
significant overheads and it's not clear how it would help in this
specific instance.

>Anyways, one of the things which cannot be a big factor is reading
>bsd.ports.mk from hard disk.

Reading 210KB from a modern disk is quite fast.

> It is certainly cached in memory when you
>run make index.

Which is the specific case being discussed in this thread.

> On the other hand it is so big that it probably doesn't
>fit in cache, or probably only fits in caches of machines generously
>endowed.

It may not stay in L1 cache but unless your system is extremely memory
stressed, it should stay in RAM.  I'd suggest that if bsd.ports.mk is
not cached then you have more serious problems to address before you
worry about the performance of make(1).

>(i am thinking Pentium 4 versus Core 2 Duo) is striking. It is less
>than 10 minutes on the big cache machine versus 30 minutes on the small
>cache one. If the cache effect is indeed dominant, then reducing the
>size of bsd.ports.mk by all possible means would be very beneficial.

This is likely to be more a combination of the crippled pentium 4 core
and a largish memory footprint for make than anything to do with
bsd.ports.mk itself.

>By the way an alternative system would be to use something other than
>make to do the job.

This is a possibility but it needs to be something that can be part
of the base system.  This means you are limited to C/C++, awk, sh
and make.  In particular, python, ruby, perl and similar scripting
languages are out.

Whilst make seems the obvious choice for the ports infrastructure, in
reality, the infrastructure does not really need or use the sort of
implicit dependency tracking and target transformations that make
excels at.  Of course, any alternative to make needs to provide a
language for defining dependencies that is equally powerful and easy
to use.

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20070529/55cf0758/attachment.pgp