svn commit: r268020 - user/nwhitehorn/condorports
Nathan Whitehorn
nwhitehorn at FreeBSD.org
Sun Jun 29 19:22:51 UTC 2014
Author: nwhitehorn
Date: Sun Jun 29 19:22:49 2014
New Revision: 268020
URL: http://svnweb.freebsd.org/changeset/base/268020
Log:
The ports tree does a number of things to configure itself to the installed
system, some of which (like testing the capabilities of /usr/bin/cc) don't
work in the context of building a dependency graph on the job submit node.
Instead, ship graph generation to the worker nodes and run it in the same
chroot as everything else.
This reduces the ports tree operations on the submit node to just distfile
fetching.
Added:
user/nwhitehorn/condorports/build_dag_dep_list.py (contents, props changed)
user/nwhitehorn/condorports/builddag.sub
user/nwhitehorn/condorports/builddagwrapper.sh (contents, props changed)
user/nwhitehorn/condorports/startbuild.sh (contents, props changed)
Modified:
user/nwhitehorn/condorports/README
user/nwhitehorn/condorports/buildportsdag.sh
user/nwhitehorn/condorports/stagebuildfiles.sh
Modified: user/nwhitehorn/condorports/README
==============================================================================
--- user/nwhitehorn/condorports/README Sun Jun 29 18:54:41 2014 (r268019)
+++ user/nwhitehorn/condorports/README Sun Jun 29 19:22:49 2014 (r268020)
@@ -24,6 +24,8 @@ Submit node prerequisites:
- sysutils/condor, configured to allow job submission
- Copies of the relevant ports trees, at the same paths as they are on the
build nodes (in /buildshare/ports/XXX)
+- python
+- A lot of disk space (> 100 GB)
Build node prerequisites:
-------------------------
@@ -41,45 +43,17 @@ Using:
The basic procedure is:
cd condorports
-./buildportsdag.sh portsdir
-condor_submit_dag -maxpre 10 ports.dag
+./startbuild.sh 355755 /path/to/output amd64 10.0-RELEASE </scratch/horde>
<Eat one to several meals depending on cluster size>
-This will configure a Condor DAG for the ports tree in
-/buildshare/ports/portsdir, the current system's architecture (see options
-below), and release (again), placing the packages in the current directory
-(once more). Submitting the DAG to the scheduler will begin the build.
-
-NOTE: The -maxpre option is *very* important, as this controls the number of
-simultaneous make fetch operations to run on the submit node. If you don't
-set it, you will get thousands of fetches at once.
-
-As the build runs, log files will appear in logs and built packages in a
-directory by default in condorports. The build queue can be inspected using
-condor_q.
-
-Options to buildportsdag.sh are specified using environment variables:
-- PORTSDIR: Name of ports directory to build in. This *MUST* occur on the same
- path on both the build and submit nodes. This directory may be read only.
- Default: NONE
-- PKGSDIR: Name of directory on submit node in which to place built packages.
- Need not exist on the build nodes and can be on a local disk.
- Default: $PORTSDIR/packages
-- DISTHORDE: Name of directory on submit node in which to store and find
- distfiles. Can be shared across ports tree. Need not exist on the build nodes
- and can be on a local disk. This directory must be writable by the user
- running the job submission.
- Default: $PORTSDIR/distfiles
-- ARCH: Value of uname -p for which to build
- Default: Value of uname -p on submit node
-- RELEASE: Name of FreeBSD release to use. This directory must exist on the
- build nodes (it need not be accessible from the submit node) in
- /releases/$RELEASE
- Default: $ARCH/`uname -r`
-- STAGEDIR: Name of directory in which to stage files for transfer for this
- particular build. Must be unique to this particular package build run and
- will be deleted at completion.
- Default: $(pwd)/stage-$ARCH/$PKGSDIR
+This will configure and submit a Condor DAG for the ports tree in
+/buildshare/ports/355755, the amd64 architecture, and the 10.0-RELEASE world,
+using the common distfiles directory in /scratch/horde (on the submit node).
+The last argument is optional. If unspecified, it will instead use a
+build-private directory in /path/to/output.
+
+As the build runs, log files and built packages will appear under
+/path/to/output. The build queue can be inspected using condor_q -wide.
TODO:
----
Added: user/nwhitehorn/condorports/build_dag_dep_list.py
==============================================================================
--- /dev/null 00:00:00 1970 (empty, because file is newly added)
+++ user/nwhitehorn/condorports/build_dag_dep_list.py Sun Jun 29 19:22:49 2014 (r268020)
@@ -0,0 +1,50 @@
+#!/usr/bin/env python
+
+import sys, os
+
+dag = file(sys.argv[1], 'r')
+dag = dag.readlines()
+
+jobs = {}
+pkgs = {}
+for line in dag:
+ line = line.split(' ')
+ if line[0] == 'JOB':
+ jobs[line[1].strip()] = []
+ if line[0] == 'PARENT':
+ deps = []
+ for dep in line[1:]:
+ if dep == 'CHILD':
+ break
+ deps.append(dep)
+ jobs[line[-1].strip()] += deps
+ if line[0] == 'VARS':
+ kvpair = line[2].split('=')
+ if kvpair[0] == 'pkg':
+ pkgs[line[1]] = kvpair[1].strip().strip('"')
+ if kvpair[0] == 'pkgdir':
+ pkgs[line[1]] = kvpair[1].strip().strip('"') + '/' + pkgs[line[1]]
+
+def deepdeps(job, deps=None):
+ if deps is None:
+ deps = []
+ for dep in jobs[job]:
+ pkg = pkgs[dep]
+ if pkg in deps:
+ continue
+ deps.append(pkg)
+ deepdeps(dep, deps)
+ return deps
+
+outdag = []
+for line in dag:
+ sline = line.split(' ')
+ if sline[0] == 'SCRIPT' and sline[1] == 'PRE':
+ line = line.strip()
+ line += ' '
+ line += ' '.join(deepdeps(sline[2]))
+ line += '\n'
+ outdag.append(line)
+
+dagout = file(sys.argv[1], 'w')
+dagout.writelines(outdag)
Added: user/nwhitehorn/condorports/builddag.sub
==============================================================================
--- /dev/null 00:00:00 1970 (empty, because file is newly added)
+++ user/nwhitehorn/condorports/builddag.sub Sun Jun 29 19:22:49 2014 (r268020)
@@ -0,0 +1,19 @@
+executable = builddagwrapper.sh
+universe = vanilla
+notification = never # failure?
+
+# Require that the job run on a machine that supports our build architecture
+# The "potemkin" is to suppress Condor's implicit Arch requirement
+requirements = Arch=="potemkin" || stringListMember("$(BuildArch)", TARGET.SupportedABIs)
+
+log=/tmp/condorlog
+output=logs/dagbuild.out
+error=logs/dagbuild.err
+
+should_transfer_files=YES
+when_to_transfer_output=ON_EXIT
+transfer_output_files=ports.dag
+transfer_input_files=buildportsdag.sh
+
+arguments= $(release_tarball) $(ports) ports.dag $(distfiles) $(outpkgs) $(stage) $(release)
+queue
Added: user/nwhitehorn/condorports/builddagwrapper.sh
==============================================================================
--- /dev/null 00:00:00 1970 (empty, because file is newly added)
+++ user/nwhitehorn/condorports/builddagwrapper.sh Sun Jun 29 19:22:49 2014 (r268020)
@@ -0,0 +1,17 @@
+#!/bin/sh
+
+export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin
+
+release_tarball=$1
+shift
+
+portsdir=$1
+
+set -e
+sudo /pkgscripts/prepbuildjail $_CONDOR_SLOT $release_tarball $portsdir
+cat buildportsdag.sh | sudo jexec $_CONDOR_SLOT sh -c 'cat > /buildportsdag.sh'
+sudo jexec $_CONDOR_SLOT sh /buildportsdag.sh $@
+sudo jexec $_CONDOR_SLOT cat /ports.dag > ports.dag
+sudo /pkgscripts/reapbuildjail $_CONDOR_SLOT
+set +e
+
Modified: user/nwhitehorn/condorports/buildportsdag.sh
==============================================================================
--- user/nwhitehorn/condorports/buildportsdag.sh Sun Jun 29 18:54:41 2014 (r268019)
+++ user/nwhitehorn/condorports/buildportsdag.sh Sun Jun 29 19:22:49 2014 (r268020)
@@ -1,28 +1,22 @@
#!/bin/sh
-# Example:
-# DISTHORDE=/scratch/horde ARCH=powerpc RELEASE=10.0-RELEASE PKGSDIR=/scratch/output/powerpc ./buildportsdag.sh 355755 ports-ppc.dag
-
export PORTSDIR=/buildshare/ports/$1
-: ${PKGSDIR=$PORTSDIR/packages}
-: ${DISTHORDE=$PORTSDIR/distfiles}
-: ${ARCH=`uname -p`}
-: ${RELEASE=`uname -r`}
-: ${STAGEDIR=$(pwd)/stage-$ARCH/$PKGSDIR}
-: ${__MAKE_CONF=/dev/null}; export __MAKE_CONF
+ARCH=`uname -p`
-export PORT_DBDIR=/var/empty # Avoid host contamination
-export PACKAGE_BUILDING=1 # Some things have different dependencies
OUTDAG=$2
PORTSET=$1
+DISTHORDE=$3
+PKGSDIR=$4
+STAGEDIR=$5
+RELEASE=$6
+OSVERSION=$(make -C $PORTSDIR -V OSVERSION)
release_tarball=$ARCH-`echo $RELEASE | tr . -`
+export PACKAGE_BUILDING=1 # Some things have different dependencies
mkdir -p $STAGEDIR
-export OSVERSION=$(tar xOf /buildshare/releases/$release_tarball.tar usr/include/sys/param.h | awk '/^\#define[[:blank:]]__FreeBSD_version/ {print $3}')
-
dagjobs=$(pwd)/ports.dagjobs
dagdeps=$(pwd)/ports.dagdeps
dag=$(pwd)/$OUTDAG
@@ -34,9 +28,9 @@ for category in $(make -C $PORTSDIR -V S
echo JOB $job package.sub >> $dagjobs
# One fetch for all archs with dummy fetch job?
- export UNAME_p=$ARCH
PKGNAME=$(make package-name)
TARBALL=$STAGEDIR/stage-$category-$port.tar
+ # Note: these are not the complete arguments to stagebuildfiles.sh. These will be filled in by build_full_deps.py
echo SCRIPT PRE $job stagebuildfiles.sh $job $PKGSDIR $ARCH $OSVERSION $TARBALL $DISTHORDE $PORTSDIR >> $dagjobs
echo SCRIPT POST $job postbuild.sh $TARBALL \$RETURN >> $dagjobs
echo VARS $job BuildArch=\"$ARCH\" >> $dagjobs
@@ -53,5 +47,6 @@ for category in $(make -C $PORTSDIR -V S
done; done
cat $dagjobs $dagdeps > $dag
-echo JOBSTATE_LOG $dag.joblog >> $dag
+echo JOBSTATE_LOG $(basename $dag).joblog >> $dag
+echo NODE_STATUS_FILE $(basename $dag).status >> $dag
rm -f $dagjobs $dagdeps
Modified: user/nwhitehorn/condorports/stagebuildfiles.sh
==============================================================================
--- user/nwhitehorn/condorports/stagebuildfiles.sh Sun Jun 29 18:54:41 2014 (r268019)
+++ user/nwhitehorn/condorports/stagebuildfiles.sh Sun Jun 29 19:22:49 2014 (r268020)
@@ -4,8 +4,10 @@
# dependent packages?
mkdir -p $(dirname $5)
+mkdir -p $2
+logfile=$5.log
exec 2>&1
-exec 1>$5.log
+exec 1>$logfile
set -e
export UNAME_p=$3
@@ -19,9 +21,10 @@ export PACKAGE_BUILDING=1
export LOCALBASE=/var/empty
# Fetch distfiles if unfetched
-cd $1
-make -d e -DBATCH checksum DISTDIR=$DISTHORDE CLEAN_FETCH_ENV=1
-make -d e -DBATCH package-links PACKAGES=$2 CLEAN_FETCH_ENV=1
+echo 'Fetching distfiles'
+make -C $1 -d e -DBATCH checksum DISTDIR=$DISTHORDE CLEAN_FETCH_ENV=1
+echo 'Making links'
+make -C $1 -d e -DBATCH package-links PACKAGES=$(pwd)/$2 CLEAN_FETCH_ENV=1
mkdir -p $2/All
# Make a tarball containing all the bits we need to build: the distfiles,
@@ -32,24 +35,25 @@ TARBALL=$5
mkdir -p $(dirname $TARBALL)
# Find distfiles
-DIST_SUBDIR=$DISTHORDE/$(make -V DIST_SUBDIR)
-ALLFILES=$(for f in `make -V ALLFILES`; do echo $DIST_SUBDIR/$f; done)
+dist_subdir=$(make -C $1 -V DIST_SUBDIR)
+DIST_SUBDIR=$DISTHORDE/$dist_subdir
+ALLFILES=$(for f in `make -C $1 -V ALLFILES`; do echo $DIST_SUBDIR/$f; done)
# Find packages
-DEPENDS=`make all-depends-list`
-PKGLIST=$(for x in $DEPENDS; do make -C $x PACKAGES=$2 -V PKGFILE; done)
+shift 7
+PKGLIST="$@"
rm -rf $TARBALL-dir
-mkdir -p $TARBALL-dir/packages $TARBALL-dir/distfiles/$(make -V DIST_SUBDIR)
+mkdir -p $TARBALL-dir/packages $TARBALL-dir/distfiles/$dist_subdir
if [ -n "$ALLFILES" ]; then
- ln -s $ALLFILES $TARBALL-dir/distfiles/$(make -V DIST_SUBDIR)
+ ln -s $ALLFILES $TARBALL-dir/distfiles/$dist_subdir
fi
if [ -n "$PKGLIST" ]; then
- ln -s $PKGLIST $TARBALL-dir/packages
+ ln $PKGLIST $TARBALL-dir/packages
fi
echo 'Building tarball'
tar -c -C $TARBALL-dir -v -L -f $TARBALL distfiles packages
rm -rf $TARBALL-dir
set +e
-rm $5.log
+rm $logfile
Added: user/nwhitehorn/condorports/startbuild.sh
==============================================================================
--- /dev/null 00:00:00 1970 (empty, because file is newly added)
+++ user/nwhitehorn/condorports/startbuild.sh Sun Jun 29 19:22:49 2014 (r268020)
@@ -0,0 +1,40 @@
+# Example:
+# startbuild.sh 355755 /path/to/build/area amd64 buildrelease [/path/to/distfiles]
+
+ports=$1
+buildpath=$2
+buildarch=$3
+buildrelease=$4
+distfiles=$5
+
+release_tarball=$buildarch-`echo $buildrelease | tr . -`
+
+if [ -z "$distfiles" ]; then
+ distfiles=$buildpath/distfiles
+fi
+
+mkdir -p $buildpath
+mkdir -p $buildpath/logs
+mkdir -p $distfiles
+
+cp builddag.sub package.sub $buildpath
+cp buildport.sh builddagwrapper.sh buildportsdag.sh stagebuildfiles.sh postbuild.sh $buildpath
+cp build_dag_dep_list.py $buildpath
+
+# First, build the real DAG
+echo JOB builddag builddag.sub > $buildpath/build.dag
+echo VARS builddag BuildArch=\"$buildarch\" >> $buildpath/build.dag
+echo VARS builddag ports=\"$ports\" >> $buildpath/build.dag
+echo VARS builddag distfiles=\"$distfiles\" >> $buildpath/build.dag
+echo VARS builddag outpkgs=\"packages\" >> $buildpath/build.dag
+echo VARS builddag stage=\"stage\" >> $buildpath/build.dag
+echo VARS builddag release_tarball=\"$release_tarball\" >> $buildpath/build.dag
+echo VARS builddag release=\"$buildrelease\" >> $buildpath/build.dag
+echo SCRIPT POST builddag build_dag_dep_list.py ports.dag >> $buildpath/build.dag
+
+# Then, run it
+echo SUBDAG EXTERNAL ports ports.dag >> $buildpath/build.dag
+echo PARENT builddag CHILD ports >> $buildpath/build.dag
+
+cd $buildpath
+condor_submit_dag -maxpre 15 -allowversionmismatch build.dag
More information about the svn-src-user
mailing list