svn commit: r258848 - user/hrs/releng/release
Hiroki Sato
hrs at FreeBSD.org
Mon Dec 2 14:40:26 UTC 2013
Author: hrs
Date: Mon Dec 2 14:40:26 2013
New Revision: 258848
URL: http://svnweb.freebsd.org/changeset/base/258848
Log:
Improve robustness and consistency, and remove redundant lines with no
major functional change:
- Use stderr for critical error messages.
- Set LC_ALL=en_US.UTF-8 for svn and LC_ALL=C for the others just in case.
This is because svn co can fail when LANG=C.
- s/freebsd/FreeBSD/.
- Use /sbin:/bin:/usr/sbin:/usr/bin instead of /bin:/sbin... for $PATH.
- Do not use /usr/local/bin in $PATH except for docproj build.
- Use $(( )) instead of $(expr).
- Do not override CHROOTDIR, SVNROOT, SRCBRANCH, DOCBRANCH, PORTBRANCH, etc.
This script should be invoked with env(1) -i to strictly control variable
definitions.
- Remove a CHROOTDIR check because CHROOTDIR is always set to the default
value (/scratch). Check existence of the directory instead.
- Define DESTDIR as /R/ explicitly.
- Use an error handler in sh(1) for ENOENT of the configuration file.
- Handle YES/NO variables for make release consistently.
- Use consistent svn co options for all trees.
- Add NODVD and NOPKG support.
- Prevent the case of !NODOC && NOPORTS for the moment. This was supported
in the old Makefile.inc.docports but it was removed.
- Do not define the default kernel to be built as GENERIC because some
architectures do not use it (GENERIC64 for powerpc64, for instance).
Let Makefile to determine the default kernel if KERNEL is empty.
Also, add KERNELS_BASE support because it was used in the old
configuration to specify the kernel. KERNEL and KERNELS_BASE should be
deprecated since KERNCONF can be used directly.
- Support the case when either TARGET or TARGET_ARCH is defined.
- Use DB_FROM_SRC for buildworld/installworld stage before entering the
chroot environment. This prevents "Required user XXX is missing" error
when building a newer system on an old system.
- Split fetch stage of the docproj port and put it just after checking out
the source trees. This eliminates network access from the chroot
environment and makes it easy to support complex network configurations.
- Add HTTP_PROXY support. It is used for subversion and make fetch-recursive
stage.
- Define a fake LOCALBASE and set CLEAN_FETCH_ENV to prevent from using
undeterministic configuration in the host environment.
DISABLE_VULNERABILITIES is also set just in case.
- Use MASTER_SITE_FREEBSD to use ftp.FreeBSD.org first.
- Unset options of X11 and SVN in docproj port build. They are not necessary.
- Always create a rtld hints file. Use forcestart instead of forcerestart.
- Use env(1) -i flag to prevent leak of variables wherever possible.
- Create empty make.conf and src.conf when it is not specified to simplify
the logic.
- Remove RELSTRING because it is no longer used.
- Use -C /path for make consistently instead of using both cd and -C.
- Remove eval for make of final release building. It was used because
KERNCONF can have multiple values, but += can handle that case better.
- TODO:
* svn co of the ports tree should be minimized to only ones needed
for doc build when !NODOC && NOPORTS. Extracting whole of the ports tree
is time-consuming on slow architectures.
* Package fetching for DVD should be done before entering chroot
environment.
Tested by: allbsd.org daily snapshot
Modified:
user/hrs/releng/release/release.sh
Modified: user/hrs/releng/release/release.sh
==============================================================================
--- user/hrs/releng/release/release.sh Mon Dec 2 14:28:30 2013 (r258847)
+++ user/hrs/releng/release/release.sh Mon Dec 2 14:40:26 2013 (r258848)
@@ -31,179 +31,206 @@
#
# $FreeBSD$
#
-
-PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin"
-export PATH
-
+set -e # Everything must succeed
+PATH="/sbin:/bin:/usr/sbin:/usr/bin"; export PATH
+LC_ALL=C; export LC_ALL
+for X in /usr/bin/svnlite /usr/local/bin/svn; do
+ [ -x $X ] && break
+done
+: ${SVN_CMD:=$X}
+: ${SVN_ARGS:=}
+: ${SH:=/bin/sh}
+: ${SYSCTL:=/sbin/sysctl}
+: ${MKDIR:=/bin/mkdir -p}
+: ${MAKE:=/usr/bin/make}
+: ${CP:=/bin/cp}
+: ${MOUNT:=/sbin/mount}
+: ${UMOUNT:=/sbin/umount}
+: ${UNAME:=/usr/bin/uname}
+: ${ID:=/usr/bin/id}
+if [ $($ID -u) -ne 0 ]; then
+ echo 1>&2 "$0: Needs to be run as root."
+ exit 1
+fi
# The directory within which the release will be built.
-CHROOTDIR="/scratch"
+: ${CHROOTDIR:=/scratch}
+: ${CHROOT_CMD:=chroot $CHROOTDIR}
+: ${DESTDIR=:/R/}
+: ${MAKEOBJDIRPREFIX:=/usr/obj}
# The default svn checkout server, and svn branches for src/, doc/,
# and ports/.
-SVNROOT="svn://svn.freebsd.org"
-SRCBRANCH="base/head at rHEAD"
-DOCBRANCH="doc/head at rHEAD"
-PORTBRANCH="ports/head at rHEAD"
+: ${SVNROOT:=svn://svn.FreeBSD.org}
+: ${SRCBRANCH:=base/head at rHEAD}
+: ${DOCBRANCH:=doc/head at rHEAD}
+: ${PORTBRANCH:=ports/head at rHEAD}
# Sometimes one needs to checkout src with --force svn option.
# If custom kernel configs copied to src tree before checkout, e.g.
-SRC_FORCE_CHECKOUT=
-
-# The default make.conf and src.conf to use. Set to /dev/null
-# by default to avoid polluting the chroot(8) environment with
-# non-default settings.
-MAKE_CONF="/dev/null"
-SRC_CONF="/dev/null"
+: ${SRC_FORCE_CHECKOUT=}
+# HTTP Proxy option. Note that http:// must be used for subversion to use
+# a proxy.
+if [ -n "$HTTP_PROXY" ]; then
+ _host=${HTTP_PROXY:%:*}
+ _port=${HTTP_PROXY:#*:}
+ case $_port in
+ //*) _host=$HTTP_PROXY; _port="" ;;
+ esac
+ SVN_ARGS="$SVN_ARGS \
+ --config-option=servers:global:http-proxy-host=$_host \
+ --config-option=servers:global:http-proxy-port=$_port \
+ "
+fi
# The number of make(1) jobs, defaults to the number of CPUs available for
# buildworld, and half of number of CPUs available for buildkernel.
-NCPU=$(sysctl -n hw.ncpu)
-if [ ${NCPU} -gt 1 ]; then
- WORLD_FLAGS="-j${NCPU}"
- KERNEL_FLAGS="-j$(expr ${NCPU} / 2)"
-fi
-MAKE_FLAGS="-s"
-
-# The name of the kernel to build, defaults to GENERIC.
-KERNEL="GENERIC"
-
-# Set to non-empty value to disable checkout of doc/ and/or ports/. Disabling
-# ports/ checkout also forces NODOC to be set.
-NODOC=
-NOPORTS=
-
-usage() {
- echo "Usage: $0 [-c release.conf]"
- exit 1
-}
+: ${NCPU:=$($SYSCTL -n hw.ncpu)}
+: ${WNCPU:=$(($NCPU + 0))} # for buildworld
+: ${KNCPU:=$((($NCPU + 1) / 2))} # for buildkernel
+: ${WORLD_FLAGS=-j$WNCPU} # can be unset
+: ${KERNEL_FLAGS=-j$KNCPU} # can be unset
+: ${MAKE_FLAGS=-s} # can be unset
while getopts c: opt; do
- case ${opt} in
+ case $opt in
c)
- RELEASECONF="${OPTARG}"
- if [ ! -e "${RELEASECONF}" ]; then
- echo "ERROR: Configuration file ${RELEASECONF} does not exist."
- exit 1
- fi
# Source the specified configuration file for overrides
- . ${RELEASECONF}
+ . "$OPTARG"
;;
\?)
- usage
+ echo 1>&2 "Usage: $0 [-c release.conf]"
+ exit 1
;;
esac
done
shift $(($OPTIND - 1))
-# If PORTS is set and NODOC is unset, force NODOC=yes because the ports tree
-# is required to build the documentation set.
-if [ "x${NOPORTS}" != "x" ] && [ "x${NODOC}" = "x" ]; then
- echo "*** NOTICE: Setting NODOC=1 since ports tree is required"
- echo " and NOPORTS is set."
- NODOC=yes
-fi
-
-# If NOPORTS and/or NODOC are unset, they must not pass to make as variables.
-# The release makefile verifies definedness of NOPORTS/NODOC variables
-# instead of their values.
-DOCPORTS=
-if [ "x${NOPORTS}" != "x" ]; then
- DOCPORTS="NOPORTS=yes "
-fi
-if [ "x${NODOC}" != "x" ]; then
- DOCPORTS="${DOCPORTS}NODOC=yes"
-fi
+# Construct release make args. The value will be normalized to true or false.
+RMAKE_ARGS_LIST="NODOC NOPKG NOPORTS NODVD"
+RMAKE_ARGS=
+for A in $RMAKE_ARGS_LIST; do
+ case $(eval echo \${$A:-no}) in
+ [Nn][Oo])
+ eval $A=false
+ ;;
+ *)
+ eval $A=true
+ RMAKE_ARGS="$RMAKE_ARGS $A=true"
+ ;;
+ esac
+done
+# Sanity check.
+# XXX: This false:true case must be supported.
+case ${NODOC}:${NOPORTS} in
+false:true)
+ echo 1>&2 "$0: NODOC is required when NOPORTS is defined."
+ exit 1
+;;
+esac
# The aggregated build-time flags based upon variables defined within
# this file, unless overridden by release.conf. In most cases, these
# will not need to be changed.
-CONF_FILES="__MAKE_CONF=${MAKE_CONF} SRCCONF=${SRC_CONF}"
-if [ "x${TARGET}" != "x" ] && [ "x${TARGET_ARCH}" != "x" ]; then
- ARCH_FLAGS="TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH}"
-else
- ARCH_FLAGS=
-fi
-CHROOT_WMAKEFLAGS="${MAKE_FLAGS} ${WORLD_FLAGS} ${CONF_FILES}"
-CHROOT_IMAKEFLAGS="${CONF_FILES}"
-CHROOT_DMAKEFLAGS="${CONF_FILES}"
-RELEASE_WMAKEFLAGS="${MAKE_FLAGS} ${WORLD_FLAGS} ${ARCH_FLAGS} ${CONF_FILES}"
-RELEASE_KMAKEFLAGS="${MAKE_FLAGS} ${KERNEL_FLAGS} KERNCONF=\"${KERNEL}\" ${ARCH_FLAGS} ${CONF_FILES}"
-RELEASE_RMAKEFLAGS="${ARCH_FLAGS} KERNCONF=\"${KERNEL}\" ${CONF_FILES} \
- ${DOCPORTS}"
-
-# Force src checkout if configured
-FORCE_SRC_KEY=
-if [ "x${SRC_FORCE_CHECKOUT}" != "x" ]; then
- FORCE_SRC_KEY="--force"
-fi
+CONF_FILES="__MAKE_CONF=/dev/null SRCCONF=/dev/null"
+CROSS_FLAGS=
+if [ -n "$TARGET" ]; then
+ CROSS_FLAGS="$CROSS_FLAGS TARGET=$TARGET"
+fi
+if [ -n "$TARGET_ARCH" ]; then
+ CROSS_FLAGS="$CROSS_FLAGS TARGET_ARCH=$TARGET_ARCH"
+fi
+: ${KERNCONF:=${KERNEL:-${KERNELS_BASE}}}
+
+DOCMAKE_ARGS=" \
+ $CONF_FILES \
+ BATCH=yes \
+ DISABLE_VULNERABILITIES=yes \
+ OPTIONS_UNSET_FORCE+=X11 \
+ OPTIONS_UNSET_FORCE+=SVN \
+ OPTIONS_UNSET_FORCE+=IGOR \
+ OPTIONS_UNSET_FORCE+=FOP \
+"
+CHROOT_MAKEFLAGS="-C ${CHROOTDIR}/usr/src -DDB_FROM_SRC"
+CHROOT_WMAKEFLAGS="$CHROOT_MAKEFLAGS $MAKE_FLAGS $WORLD_FLAGS $CONF_FILES"
+CHROOT_IMAKEFLAGS="$CHROOT_MAKEFLAGS $MAKE_FLAGS $CONF_FILES"
+CHROOT_DMAKEFLAGS="$CHROOT_MAKEFLAGS $MAKE_FLAGS $CONF_FILES"
+RELEASE_WMAKEFLAGS="-C /usr/src $MAKE_FLAGS $WORLD_FLAGS $CROSS_FLAGS"
+RELEASE_KMAKEFLAGS="-C /usr/src $MAKE_FLAGS $KERNEL_FLAGS $CROSS_FLAGS"
+RELEASE_RMAKEFLAGS="-C /usr/src/release $MAKE_FLAGS $CROSS_FLAGS $RMAKE_ARGS"
+# KERNCONF can be empty to leave the default value to Makefile.
+for K in $KERNCONF; do
+ RELEASE_KMAKEFLAGS="$RELEASE_KMAKEFLAGS KERNCONF+=$K"
+ RELEASE_RMAKEFLAGS="$RELEASE_RMAKEFLAGS KERNCONF+=$K"
+done
+SETENV="env -i PATH=$PATH LC_ALL=$LC_ALL"
+SETENV_CHROOT="$SETENV MAKEOBJDIRPREFIX=$MAKEOBJDIRPREFIX"
-if [ ! ${CHROOTDIR} ]; then
- echo "Please set CHROOTDIR."
- exit 1
+# Force src checkout if configured.
+if [ -n "$SRC_FORCE_CHECKOUT" ]; then
+ SVN_ARGS="$SVN_ARGS --force"
fi
-
-if [ $(id -u) -ne 0 ]; then
- echo "Needs to be run as root."
+if [ ! -d "$CHROOTDIR" ]; then
+ echo 1>&2 "$0: $CHROOTDIR not found."
exit 1
fi
-set -e # Everything must succeed
-
-mkdir -p ${CHROOTDIR}/usr
-
-svn co ${FORCE_SRC_KEY} ${SVNROOT}/${SRCBRANCH} ${CHROOTDIR}/usr/src
-if [ "x${NODOC}" = "x" ]; then
- svn co ${SVNROOT}/${DOCBRANCH} ${CHROOTDIR}/usr/doc
-fi
-if [ "x${NOPORTS}" = "x" ]; then
- svn co ${SVNROOT}/${PORTBRANCH} ${CHROOTDIR}/usr/ports
-fi
-
-cp /etc/resolv.conf ${CHROOTDIR}/etc/resolv.conf
-cd ${CHROOTDIR}/usr/src
-make ${CHROOT_WMAKEFLAGS} buildworld
-make ${CHROOT_IMAKEFLAGS} installworld DESTDIR=${CHROOTDIR}
-make ${CHROOT_DMAKEFLAGS} distribution DESTDIR=${CHROOTDIR}
-mount -t devfs devfs ${CHROOTDIR}/dev
-trap "umount ${CHROOTDIR}/dev" EXIT # Clean up devfs mount on exit
-
-build_doc_ports() {
- # Run ldconfig(8) in the chroot directory so /var/run/ld-elf*.so.hints
- # is created. This is needed by ports-mgmt/pkg.
- chroot ${CHROOTDIR} /etc/rc.d/ldconfig forcerestart
+# Check out trees.
+$MKDIR ${CHROOTDIR}/usr
+$SETENV LC_ALL=en_US.UTF-8 $SVN_CMD co $SVN_ARGS \
+ ${SVNROOT}/${SRCBRANCH} ${CHROOTDIR}/usr/src
+if ! $NODOC; then
+ $SETENV LC_ALL=en_US.UTF-8 $SVN_CMD co $SVN_ARGS \
+ ${SVNROOT}/${DOCBRANCH} ${CHROOTDIR}/usr/doc
+fi
+if ! $NOPORTS; then
+ $SETENV LC_ALL=en_US.UTF-8 $SVN_CMD co $SVN_ARGS \
+ ${SVNROOT}/${PORTBRANCH} ${CHROOTDIR}/usr/ports
+fi
+# Fetch distfiles for ports-mgmt/pkg and textproc/docproj port if necessary.
+if ! $NODOC && ! $NOPORTS; then
+ # LOCALBASE=/var/empty is required to disable automatic detection of
+ # PERL_VERSION. It depends on existence of ${LOCALBASE}/bin/perl5.
+ # CLEAN_FETCH_ENV disables ${LOCALBASE}/sbin/pkg dependency.
+ $MKDIR ${CHROOTDIR}/usr/ports/distfiles
+ for P in ports-mgmt/pkg textproc/docproj; do
+ $SETENV PORTSDIR=${CHROOTDIR}/usr/ports \
+ HTTP_PROXY=$HTTP_PROXY \
+ $MAKE -C ${CHROOTDIR}/usr/ports/$P \
+ LOCALBASE=/var/empty \
+ CLEAN_FETCH_ENV=yes \
+ MASTER_SITE_FREEBSD=yes \
+ $DOCMAKE_ARGS fetch-recursive
+ done
+fi
+
+# Build a clean environment by using $CHROOTDIR/usr/src and install it into
+# $CHROOTDIR. Note that this is always a native build.
+$SETENV_CHROOT $MAKE $CHROOT_WMAKEFLAGS buildworld
+$SETENV_CHROOT $MAKE $CHROOT_IMAKEFLAGS installworld DESTDIR=$CHROOTDIR
+$SETENV_CHROOT $MAKE $CHROOT_DMAKEFLAGS distribution DESTDIR=$CHROOTDIR
+$MOUNT -t devfs devfs ${CHROOTDIR}/dev
+trap "$UMOUNT ${CHROOTDIR}/dev" EXIT # Clean up devfs mount on exit
+
+# If MAKE_CONF and/or SRC_CONF are set copy them to the chroot.
+# If not, create empty one.
+$CP ${MAKE_CONF:-/dev/null} ${CHROOTDIR}/etc/make.conf
+$CP ${SRC_CONF:-/dev/null} ${CHROOTDIR}/etc/src.conf
+
+# Run ldconfig(8) in the chroot directory so /var/run/ld-elf*.so.hints
+# is created. This is needed by ports-mgmt/pkg.
+$CHROOT_CMD $SETENV $SH /etc/rc.d/ldconfig forcestart
+# Build docproj port if necessary.
+if ! $NODOC && ! $NOPORTS; then
## Trick the ports 'run-autotools-fixup' target to do the right thing.
- _OSVERSION=$(sysctl -n kern.osreldate)
- if [ -d ${CHROOTDIR}/usr/doc ] && [ "x${NODOC}" = "x" ]; then
- PBUILD_FLAGS="OSVERSION=${_OSVERSION} BATCH=yes"
- PBUILD_FLAGS="${PBUILD_FLAGS}"
- chroot ${CHROOTDIR} make -C /usr/ports/textproc/docproj \
- ${PBUILD_FLAGS} OPTIONS_UNSET="FOP IGOR" install clean distclean
- fi
-}
-
-# If MAKE_CONF and/or SRC_CONF are set and not character devices (/dev/null),
-# copy them to the chroot.
-if [ -e ${MAKE_CONF} ] && [ ! -c ${MAKE_CONF} ]; then
- mkdir -p ${CHROOTDIR}/$(dirname ${MAKE_CONF})
- cp ${MAKE_CONF} ${CHROOTDIR}/${MAKE_CONF}
-fi
-if [ -e ${SRC_CONF} ] && [ ! -c ${SRC_CONF} ]; then
- mkdir -p ${CHROOTDIR}/$(dirname ${SRC_CONF})
- cp ${SRC_CONF} ${CHROOTDIR}/${SRC_CONF}
-fi
-
-if [ -d ${CHROOTDIR}/usr/ports ]; then
- build_doc_ports ${CHROOTDIR}
-fi
-
-if [ "x${RELSTRING}" = "x" ]; then
- RELSTRING="$(chroot ${CHROOTDIR} uname -s)-${OSRELEASE}-${TARGET_ARCH}"
-fi
-
-eval chroot ${CHROOTDIR} make -C /usr/src ${RELEASE_WMAKEFLAGS} buildworld
-eval chroot ${CHROOTDIR} make -C /usr/src ${RELEASE_KMAKEFLAGS} buildkernel
-eval chroot ${CHROOTDIR} make -C /usr/src/release ${RELEASE_RMAKEFLAGS} \
- release RELSTRING=${RELSTRING}
-eval chroot ${CHROOTDIR} make -C /usr/src/release ${RELEASE_RMAKEFLAGS} \
- install DESTDIR=/R RELSTRING=${RELSTRING}
+ $CHROOT_CMD $SETENV PATH=${PATH}:/usr/local/bin \
+ $MAKE -C /usr/ports/textproc/docproj \
+ OSVERSION=$($SYSCTL -n kern.osreldate) \
+ $DOCMAKE_ARGS install
+fi
+# Build a release in $CHROOTDIR and install it into $DESTDIR.
+# This can be a cross build.
+$CHROOT_CMD $SETENV $MAKE $RELEASE_WMAKEFLAGS buildworld
+$CHROOT_CMD $SETENV $MAKE $RELEASE_KMAKEFLAGS buildkernel
+$CHROOT_CMD $SETENV $MAKE $RELEASE_RMAKEFLAGS release
+$CHROOT_CMD $SETENV $MAKE $RELEASE_RMAKEFLAGS install DESTDIR=$DESTDIR
More information about the svn-src-user
mailing list