Fully parallelized "make universe"

[Ruslan Ermilov]

Poul-Henning Kamp wrote:
> David O'Brien writes:
> >Actually I'd really love for a way to do the outter loop in parallel.
> >Due to -j messing up error output too much, I'd like to have the work for
> >each platform going in parallel instead.  E.g.:
> >
> >    make universe for i386 only &
> >    make universe for powerpc only &
> >    make universe for sparc64 only &
> >    make universe for amd64 only &
> >    make universe for alpha only &
> >    make universe for ia64 only &
> 
> True, but I can't get that working no matter how much I've tried.
> 
OK, here it comes.  The attached patch unrolls the main universe
loop, making it possible for "universe" subtargets corresponding
to individual architectures to be built in parallel.

EXAMPLES

make -j4 universe
	will build four architectures in parallel, each will be
	built using four parallel make's.

make -j8 universe JFLAG=-j4
	will build eight architectures in parallel, each will be
	built using four parallel make's.

make -j8 universe JFLAG=-B
	will build eight architectures in parallel, each will be
	built using one make.

Please test.

Points considered: since "buildkernels" is implemented using the
make's .for loop, kernels for a given architecture are built
sequentially (each can still be built in parallel).  I didn't see
much point in parallelizing this task, but this is easily doable.
(One redundant JFLAG when calling buildkernels was removed by
the patch.)


Cheers,
-- 
Ruslan Ermilov
ru at FreeBSD.org
FreeBSD committer
-------------- next part --------------
Index: Makefile
===================================================================
RCS file: /home/ncvs/src/Makefile,v
retrieving revision 1.306
diff -u -r1.306 Makefile
--- Makefile	9 Aug 2004 11:38:41 -0000	1.306
+++ Makefile	17 Aug 2004 07:54:43 -0000
@@ -144,10 +144,9 @@
 # Set a reasonable default
 .MAIN:	all
 
-.if defined(HISTORICAL_MAKE_WORLD) || defined(DESTDIR)
-.if make(world)
 STARTTIME!= LC_ALL=C date
-.endif
+
+.if defined(HISTORICAL_MAKE_WORLD) || defined(DESTDIR)
 #
 # world
 #
@@ -242,16 +241,20 @@
 # universe
 #
 # Attempt to rebuild *everything* for all supported architectures,
-# with reasonable chance of success, regardless of how old your
+# with a reasonable chance of success, regardless of how old your
 # existing system is.
 #
-i386_mach=	pc98
-universe:
+universe: universe_prologue
+universe_prologue:
 	@echo "--------------------------------------------------------------"
 	@echo ">>> make universe started on ${STARTTIME}"
 	@echo "--------------------------------------------------------------"
-.for arch in i386 sparc64 alpha ia64 amd64
-.for mach in ${arch} ${${arch}_mach}
+.for target in i386 i386:pc98 sparc64 alpha ia64 amd64
+.for arch in ${target:C/:.*$//}
+.for mach in ${target:C/^.*://}
+universe: universe_${mach}
+.ORDER: universe_prologue universe_${mach} universe_epilogue
+universe_${mach}:
 	@echo ">> ${mach} started on `LC_ALL=C date`"
 	-cd ${.CURDIR} && ${MAKE} ${JFLAG} buildworld \
 	    TARGET_ARCH=${arch} TARGET=${mach} \
@@ -262,10 +265,13 @@
 	-cd ${.CURDIR}/sys/${mach}/conf && ${MAKE} LINT \
 	    > ${.CURDIR}/_.${mach}.makeLINT 2>&1
 .endif
-	cd ${.CURDIR} && ${MAKE} ${JFLAG} buildkernels TARGET_ARCH=${arch} TARGET=${mach}
+	cd ${.CURDIR} && ${MAKE} buildkernels TARGET_ARCH=${arch} TARGET=${mach}
 	@echo ">> ${mach} completed on `LC_ALL=C date`"
 .endfor
 .endfor
+.endfor
+universe: universe_epilogue
+universe_epilogue:
 	@echo "--------------------------------------------------------------"
 	@echo ">>> make universe completed on `LC_ALL=C date`"
 	@echo "                      (started ${STARTTIME})"
@@ -274,8 +280,6 @@
 .if make(buildkernels)
 KERNCONFS!=	cd ${.CURDIR}/sys/${TARGET}/conf && \
 		find [A-Z]*[A-Z] -type f -maxdepth 0 ! -name NOTES
-.endif
-
 buildkernels:
 .for kernel in ${KERNCONFS}
 	-cd ${.CURDIR} && ${MAKE} ${JFLAG} buildkernel \
@@ -283,3 +287,4 @@
 	    __MAKE_CONF=/dev/null \
 	    > _.${TARGET}.${kernel} 2>&1
 .endfor
+.endif
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-current/attachments/20040817/22620dde/attachment.bin