ports/113132: Allow -jx for port builds

Tue May 29 17:10:05 UTC 2007

>Number:         113132
>Category:       ports
>Synopsis:       Allow -jx for port builds
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-ports-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Tue May 29 17:10:04 GMT 2007
>Closed-Date:
>Last-Modified:
>Originator:     Benjamin Lutz
>Release:        FreeBSD 6.2-RELEASE-p5 i386
>Organization:
>Environment:
System: FreeBSD atlantis.intranet 6.2-RELEASE-p5 FreeBSD 6.2-RELEASE-p5 #8: Wed May 23 22:09:01 CEST 2007 maxlor at atlantis.intranet:/usr/obj/usr/src/sys/ATLANTIS i386


>Description:

This patch allows ports to build with make -jx on multi-CPU machines 
(where x is typically 1 + the number of CPUs in the system) in order to
speed up the build process. All a port maintainer has to do is set the
ALLOW_MAKE_JOBS variable in a port Makefile, whereas the user has to
set ENABLE_MAKE_JOBS in his /etc/make.conf. If these two variables are
set, the code in the patch will get the number of cpus, add 1, then
call make with, e.g., -j3.

Here's a description of the variables involved:

ENABLE_MAKE_JOBS: The master switch that enables or disables the whole
    thing. The user is supposed to set it in his /etc/make.conf . If
    this variable isn't set, the ports build like they always did.

ALLOW_MAKE_JOBS: Goes into a port's makefile. The port maintainer
    indicates with it that the port can be built with multiple make
    jobs.

MAKE_JOBS_WHITELIST: Allows the user to override ALLOW_MAKE_JOBS. Any
    port whose UNIQUENAME is listed in MAKE_JOBS_WHITELIST will have
    its ALLOW_MAKE_JOBS defined. The user would put something like this
    in his /etc/make.conf: MAKE_JOBS_WHITELIST=kdebase gtk20

The following are new internal variables I introduced:

CPUS: The number of CPUs in the system.

MAKE_JOBS_NJOBS: The number of make jobs that will be used. Currently,
    this is ${CPUS} + 1, which benchmarks have shown to be the most
    efficient number on my dual-core system.

MAKE_JOBS_ARGS: The argument that is passed to make.

BUILD_FAILMSG: A message that is printed if the do-build target fails. 
    Note that this variable can be used by any part of the ports system,
    not just the MAKE_JOBS part. To use it, write code like

      BUILD_FAILMSG+= "Hello World"

    Each message added to BUILD_FAILMSG like this will be printed at the
    end of the do-build stage (if it fails), one paragraph per message.

    The MAKE_JOBS code currently uses this to inform the user that he
    needn't bother to send bug reports if MAKE_JOBS_WHITELIST is used.


I have tested this patch throughout the X.org 7.2 upgrade process with
several hundred port builds, it seems to work well; many ports seem to
support building with multiple make jobs.

>How-To-Repeat:
>Fix:

--- make_jobs.diff begins here ---
diff -ruN Mk.orig/bsd.port.mk Mk/bsd.port.mk
--- Mk.orig/bsd.port.mk	Tue May 29 18:36:31 2007
+++ Mk/bsd.port.mk	Tue May 29 18:21:16 2007
@@ -1235,6 +1235,9 @@
 .endif
 .endif
 
+# Get the number of CPUs
+CPUS!=	${SYSCTL} -n kern.smp.cpus
+
 MASTERDIR?=	${.CURDIR}
 
 .if ${MASTERDIR} != ${.CURDIR}
@@ -2255,6 +2258,27 @@
 			MOTIFLIB="${MOTIFLIB}" LIBDIR="${LIBDIR}" CFLAGS="${CFLAGS}" \
 			CXXFLAGS="${CXXFLAGS}" MANPREFIX="${MANPREFIX}"
 
+# Set up building with multiple make jobs
+.if defined(ENABLE_MAKE_JOBS)
+.if !defined(ALLOW_MAKE_JOBS) && defined(MAKE_JOBS_WHITELIST)
+ALLOW_MAKE_JOBS!=	${ECHO} " ${MAKE_JOBS_WHITELIST} " | ${GREP} -o "[[:space:]]${UNIQUENAME}[[:space:]]" || ${TRUE}
+.if ${ALLOW_MAKE_JOBS} == ""
+.undef ALLOW_MAKE_JOBS
+.else
+BUILD_FAILMSG+= "You're using MAKE_JOBS_WHITELIST to force this port to build\
+with multiple make jobs. This is not supported. Do not report this failure\
+but remove this port's UNIQUENAME (${UNIQUENAME}) from MAKE_JOBS_WHITELIST\
+and try again."
+.endif
+.endif
+.if defined(ALLOW_MAKE_JOBS)
+MAKE_JOBS_NJOBS!=	${EXPR} ${CPUS} + 1
+MAKE_JOBS_ARGS?=	-j${MAKE_JOBS_NJOBS}
+.else
+MAKE_JOBS_ARGS?=
+.endif
+.endif
+
 PTHREAD_CFLAGS?=
 PTHREAD_LIBS?=		-pthread
 
@@ -3734,12 +3758,30 @@
 .if !target(do-build)
 do-build:
 .if defined(USE_GMAKE)
-	@(cd ${BUILD_WRKSRC}; ${SETENV} ${MAKE_ENV} ${GMAKE} ${MAKE_FLAGS} ${MAKEFILE} ${MAKE_ARGS} ${ALL_TARGET})
+	@(cd ${BUILD_WRKSRC}; ${SETENV} ${MAKE_ENV} ${GMAKE} ${MAKE_FLAGS} ${MAKEFILE} ${MAKE_JOBS_ARGS} ${MAKE_ARGS} ${ALL_TARGET} || (\
+		RETVAL=$$?;\
+		if [ x != x${BUILD_FAILMSG:Q} ]; then\
+			${PRINTF} "\n" ;\
+			${PRINTF} "===> %s\n\n" ${BUILD_FAILMSG} | ${FMT} 75 79; \
+		fi;\
+		return $$RETVAL ))
 .else
 .if defined(PERL_MODBUILD)
-	@(cd ${BUILD_WRKSRC}; ${SETENV} ${MAKE_ENV} ${PERL5} ${PL_BUILD} ${MAKE_ARGS} ${ALL_TARGET})
-.else
-	@(cd ${BUILD_WRKSRC}; ${SETENV} ${MAKE_ENV} ${MAKE} ${MAKE_FLAGS} ${MAKEFILE} ${MAKE_ARGS} ${ALL_TARGET})
+	@(cd ${BUILD_WRKSRC}; ${SETENV} ${MAKE_ENV} ${PERL5} ${PL_BUILD} ${MAKE_ARGS} ${ALL_TARGET} || (\
+		RETVAL=$$?;\
+		if [ x != x${BUILD_FAILMSG:Q} ]; then\
+			${PRINTF} "\n" ;\
+			${PRINTF} "===> %s\n\n" ${BUILD_FAILMSG} | ${FMT} 75 79; \
+		fi;\
+		return $$RETVAL ))
+.else
+	@(cd ${BUILD_WRKSRC}; ${SETENV} ${MAKE_ENV} ${MAKE} ${MAKE_FLAGS} ${MAKEFILE} ${MAKE_JOBS_ARGS} ${MAKE_ARGS} ${ALL_TARGET} || (\
+		RETVAL=$$?;\
+		if [ x != x${BUILD_FAILMSG:Q} ]; then\
+			${PRINTF} "\n" ;\
+			${PRINTF} "===> %s\n\n" ${BUILD_FAILMSG} | ${FMT} 75 79; \
+		fi;\
+		return $$RETVAL ))
 .endif
 .endif
 .endif
--- make_jobs.diff ends here ---


>Release-Note:
>Audit-Trail:
>Unformatted:

