svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin

[Warner Losh]

Why not have this ‘opt out’ rather than ‘opt in’ like it is now? Are there any known bad dependencies this introduces?

Warner

On Mar 26, 2014, at 4:30 PM, Dimitry Andric <dim at freebsd.org> wrote:

> Author: dim
> Date: Wed Mar 26 22:30:38 2014
> New Revision: 263778
> URL: http://svnweb.freebsd.org/changeset/base/263778
> 
> Log:
>  Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
>  all the SUBDIR entries in parallel, instead of serially.  Apply this
>  option to a selected number of Makefiles, which can greatly speed up the
>  build on multi-core machines, when using make -j.
> 
>  This can be extended to more Makefiles later on, whenever they are
>  verified to work correctly with parallel building.
> 
>  I tested this on a 24-core machine, with make -j48 buildworld (N = 6):
> 
>                  before    stddev       after    stddev
>                  =======   ======       =======  ======
>  real time        1741.1     16.5         959.8     2.7
>  user time       12468.7     16.4       14393.0    16.8
>  sys  time        1825.0     54.8        2110.6    22.8
> 
>  (user+sys)/real     8.2                   17.1
> 
>  E.g. the build was approximately 45% faster in real time.  On machines
>  with less cores, or with lower -j settings, the speedup will not be as
>  impressive.  But at least you can now almost max out a machine with
>  buildworld!
> 
>  Submitted by:	jilles
>  MFC after:	2 weeks
> 
> Modified:
>  head/bin/Makefile
>  head/lib/Makefile
>  head/lib/clang/Makefile
>  head/sbin/Makefile
>  head/share/mk/bsd.subdir.mk
>  head/usr.bin/Makefile
>  head/usr.sbin/Makefile
> 
> Modified: head/bin/Makefile
> ==============================================================================
> --- head/bin/Makefile	Wed Mar 26 20:43:40 2014	(r263777)
> +++ head/bin/Makefile	Wed Mar 26 22:30:38 2014	(r263778)
> @@ -60,4 +60,6 @@ SUBDIR+=	tests
> 
> SUBDIR:=	${SUBDIR:O}
> 
> +SUBDIR_PARALLEL=
> +
> .include <bsd.subdir.mk>
> 
> Modified: head/lib/Makefile
> ==============================================================================
> --- head/lib/Makefile	Wed Mar 26 20:43:40 2014	(r263777)
> +++ head/lib/Makefile	Wed Mar 26 22:30:38 2014	(r263778)
> @@ -276,4 +276,8 @@ afterinstall:
> 	${INSTALL_SYMLINK} ../include ${DESTDIR}/usr/lib/include
> .endif
> 
> +.if !make(install)
> +SUBDIR_PARALLEL=
> +.endif
> +
> .include <bsd.subdir.mk>
> 
> Modified: head/lib/clang/Makefile
> ==============================================================================
> --- head/lib/clang/Makefile	Wed Mar 26 20:43:40 2014	(r263777)
> +++ head/lib/clang/Makefile	Wed Mar 26 22:30:38 2014	(r263778)
> @@ -147,4 +147,6 @@ SUBDIR+=liblldb \
> 
> SUBDIR+= include
> 
> +SUBDIR_PARALLEL=
> +
> .include <bsd.subdir.mk>
> 
> Modified: head/sbin/Makefile
> ==============================================================================
> --- head/sbin/Makefile	Wed Mar 26 20:43:40 2014	(r263777)
> +++ head/sbin/Makefile	Wed Mar 26 22:30:38 2014	(r263778)
> @@ -126,4 +126,6 @@ SUBDIR+=	tests
> 
> SUBDIR:=	${SUBDIR:O}
> 
> +SUBDIR_PARALLEL=
> +
> .include <bsd.subdir.mk>
> 
> Modified: head/share/mk/bsd.subdir.mk
> ==============================================================================
> --- head/share/mk/bsd.subdir.mk	Wed Mar 26 20:43:40 2014	(r263777)
> +++ head/share/mk/bsd.subdir.mk	Wed Mar 26 22:30:38 2014	(r263778)
> @@ -71,7 +71,26 @@ ${SUBDIR}: .PHONY .MAKE
> .for __target in all all-man checkdpadd clean cleandepend cleandir \
>     cleanilinks depend distribute lint maninstall manlint obj objlink \
>     realinstall regress tags ${SUBDIR_TARGETS}
> +.ifdef SUBDIR_PARALLEL
> +.for __dir in ${SUBDIR}
> +${__target}: ${__target}_subdir_${__dir}
> +${__target}_subdir_${__dir}: .MAKE
> +	@${_+_}set -e; \
> +		if test -d ${.CURDIR}/${__dir}.${MACHINE_ARCH}; then \
> +			${ECHODIR} "===> ${DIRPRFX}${__dir}.${MACHINE_ARCH} (${__target:realinstall=install})"; \
> +			edir=${__dir}.${MACHINE_ARCH}; \
> +			cd ${.CURDIR}/$${edir}; \
> +		else \
> +			${ECHODIR} "===> ${DIRPRFX}${__dir} (${__target:realinstall=install})"; \
> +			edir=${__dir}; \
> +			cd ${.CURDIR}/$${edir}; \
> +		fi; \
> +		${MAKE} ${__target:realinstall=install} \
> +		    DIRPRFX=${DIRPRFX}$$edir/
> +.endfor
> +.else
> ${__target}: _SUBDIR
> +.endif
> .endfor
> 
> .for __target in files includes
> 
> Modified: head/usr.bin/Makefile
> ==============================================================================
> --- head/usr.bin/Makefile	Wed Mar 26 20:43:40 2014	(r263777)
> +++ head/usr.bin/Makefile	Wed Mar 26 22:30:38 2014	(r263778)
> @@ -379,4 +379,6 @@ SUBDIR+=	svn
> 
> SUBDIR:=	${SUBDIR:O}
> 
> +SUBDIR_PARALLEL=
> +
> .include <bsd.subdir.mk>
> 
> Modified: head/usr.sbin/Makefile
> ==============================================================================
> --- head/usr.sbin/Makefile	Wed Mar 26 20:43:40 2014	(r263777)
> +++ head/usr.sbin/Makefile	Wed Mar 26 22:30:38 2014	(r263778)
> @@ -320,4 +320,6 @@ SUBDIR+=	wpa
> 
> SUBDIR:=	${SUBDIR:O}
> 
> +SUBDIR_PARALLEL=
> +
> .include <bsd.subdir.mk>
>