depend + all vs dependall

[Bruce Evans]

On Sat, 29 Mar 2003, M. Warner Losh wrote:

> NetBSD created a dependall target some time ago.  This target does a
> make depend and then a make all so they only have to traverse the tree
> once for these two stages rather than twice.  The time of a buildworld
> came up in a discussion recently and I thought I'd see how hard it
> would be to do something similar in FreeBSD.  Here are my preliminary
> results.
>
> Machine: Dell Inspiron 8000, 256M RAM, P3-700
> time make buildworld
> (2:04:34 wall time, didn't save the actual output :-(.
>
> Machine: Dual Athlon XP2000+ 1.5G RAM aac controller.
>
> time make buildworld -j 8 -s

Note that all benchmarks using -j are invalid because of nondeterministic
wait times of up to 100 msec for each job.  This pessimizes makeworld -j 4
times by about 20% on a non-dual Athlon XP1600, and can't do good things
for the variance.  The pessimization is larger on faster machines of
course.  This is fixed in NetBSD.  FreeBSD only has the hackaround of
reducing the timeout from 500 msec to 100 msec.

> run 0: did the above to 'flush the caches/load the sources in ram'
>
> Pre-change:
>
> Run 1:
> 1941.458u 723.640s 32:23.67 137.1%      2747+2215k 1447+145802io 465pf+0w
> Run 2:
> 1942.160u 729.972s 31:45.84 140.2%      2748+2212k 1423+145755io 465pf+0w

The SMP overheads seem to be very large.  I get the following times on a
non-Dual Athlon XP1600 overclocked 256MB RAM ide controller 2 drives:

%%%
--------------------------------------------------------------
>>> elf make world completed on Sun Mar  2 16:30:55 EST 2003
                       (started Sun Mar  2 15:53:15 EST 2003)
--------------------------------------------------------------
     2260.31 real      1729.55 user       326.24 sys
%%%

This machine had lost 256MB of its RAM at the time of the above benchmark
(the latest one that I have with no local changes to the src tree).
Losing 256MB cost it a 100-200 seconds.  Upgrading to 1024 MB RAM
improved on its old speed of 1967 seconds to 1943 seconds (both of
these times with local changes).  The disk cache is cold in all of my
makeworld benchmarks.  A warm cache wouldn't have much helped with
512MB RAM since that is not quite enough to cache the src tree, but
it reduces the makeworld times a little more with 1024 MB RAM.

> After Changes:
>
> Run 1:
> 1922.767u 723.847s 30:48.64 143.1%      2785+2201k 1312+148256io 465pf+0w
> Run 2:
> 1922.661u 725.477s 30:49.99 143.1%      2788+2201k 1378+148489io 465pf+0w
>
> So it looks like it saves a little over a minute out of 32 (1925s
> average vs 1849s average, or almost a 4% reduction) on my big build
> box.

It is a bug for make depend to be run at all in the default (non-NOCLEAN)
case.  My commits for this got clobbered, but I still use them here.
This seems to save only about 5% currently (down from 10% when the
changes were committed in 1998).

> My only concern with the patches is that they might interact badly
> with a bug I remember from the FreeBSD 1.1R days, but can't reproduce,
> in make.  Once upon a time, 'make depend all' was different than 'make
> depend && make all' because the .depend files weren't re-read after
> the depend phase, but before the all phase, whereas two makes this
> would be the case.  Since this change combines the two, I'm a little
> worried about that.  Is that still a bug in FreeBSD's make?  It won't
> matter for a pure, virgin tree, but might for incremental builds...

This is not a bug, but is how make works.  It shouldn't be a problem if
dependall is implemented correctly.  dependall should avoid the double
tree traversal but somehow build "depend" and "all" sequentially in
leaf directories.

> ==== //depot/user/imp/freebsd-imp/share/mk/bsd.README#3 (text+ko) ====
>
> @@ -169,6 +169,8 @@
>  	depend:
>  		make the dependencies for the source files, and store
>  		them in the file .depend.
> +	dependall:
> +		make depend then make all

This at least describes a correct implementation :-).

>  	install:
>  		install the program and its manual pages; if the Makefile
>  		does not itself define the target install, the targets
>
> ==== //depot/user/imp/freebsd-imp/share/mk/bsd.dep.mk#3 (text+ko) ====
>
> @@ -31,6 +31,9 @@
>  #		Make the dependencies for the source files, and store
>  #		them in the file ${DEPENDFILE}.
>  #
> +#	dependall:
> +#		make depend and then all
> +#

The wording is different from that in the README, and is poor in both
places.

>  #	tags:
>  #		In "ctags" mode, create a tags file for the source files.
>  #		In "gtags" mode, create a (GLOBAL) gtags file for the
> @@ -183,3 +186,7 @@
>  		echo "LDADD -> $$ldadd1" ; \
>  	fi
>  .endif
> +
> +.PHONY: dependall
> +.ORDER: afterdepend all
> +dependall: depend all

.PHONY doesn't work right with BSD make, and is not use for any of the
other phony depend targets in FreeBSD.

The dependencies seem to be correct, but I think it's a style bug to
have afterdepend in the .ORDER statement instead of "depend", at least
in FreeBSD.  afterdepend isn't actually done after "depend"; "depend"
depends on afterdepend so the latter is part of the former (this is
another style bug).

Bruce