Parallel builds, PKG_DBDIR locks
dforsyth at FreeBSD.org
Sun Mar 22 22:00:20 PDT 2009
Last year for my Summer of Code project, I worked on adding parallel build
support to ports. While it worked, some of it didn't work right, and I
wasn't really happy with the final result. I finally got some free time
lately and decided to clean it up a little bit, and since it's at a point
where I can use it without fearing cataclysmic failure, I figured I'd share
with anyone else who might be interested.
What bsd.locked.mk allows you to do first and foremost is build more than
one port at a time without fearing for your PKG_DBDIR or overwrites when
ports with common dependencies build as long as you have the USE_LOCKS flag
set. It creates a lock in a building ports directory that, if kept
consistent, will prevent the port from being built more than once at a time.
When it comes time for a port to register itself (fake-pkg), PKG_DBDIR
itself is used as a lock to prevent inadvertent clobbering of the database.
I also added a locking for fetches, so that if ports share distfiles (qt4
stuff, for instance), your builds won't overwrite the files or outright
fail. With P_DEPENDS and P_FETCH, you have the option of parallel
dependency handling and distfile fetching. P_DEPENDS=4, for instance, will
allow a port to build up to 4 dependencies at a time. P_FETCH=3 will allow
up to 3 distfiles of a port to be fetched at a time.
I've generated a patch for anyone who wants to test this out. If the patch
fails to apply cleanly, just grab the makefile and add the conditional for
USE_LOCKS right before the master-sites-ALL target. If you don't have
USE_LOCKS set, everything will work as if bsd.locked.mk isn't even there. I
had a patch from last summer that made pkg_install and pkg_deinstall honor
the locks as well, but I'm not sure where I put it, and it's probably no
good at this point anyway. It was pretty simple, regardless, and I could
redo it if anyone actually takes an interest in this.
Run make config-recursive or use BATCH if you use this or else you might be
some nasty suprises.
patch: http://bsdtips.utcorp.net/~dforsyth/bsd.locked.mk.diff (apply it in
your ports makefile directory)
More information about the freebsd-ports