Limitations of Ports System

[Erik Trulsson]

On Fri, Dec 14, 2007 at 10:34:06PM -0500, Yoshihiro Ota wrote:
> On Thu, 13 Dec 2007 21:58:57 +0100
> Erik Trulsson <ertr1013 at student.uu.se> wrote:
> 
> > One shortcoming is the lack of locking making parallell builds a bit unsafe.
> > If you try to build both port A and port B at the same time, and both A and
> > B depends (directly or indirectly) on port C which is not installed, then
> > you can esily end up having two processes both trying to build C at the same
> > time.  This usually confuses both builds very badly making them fail.
> > 
> > I also don't think there is any locking on /var/db/pkg making possibly
> > somewhat unsafe trying to register the installation of two ports/packages at
> > the same time.  I have never noticed any actual problems with this though.
> > 
> > 
> > Some sort of locking, making parallel builds safe, should be possible to
> > add to the ports system without doing any sweeping changes.
> > (I did look briefly at the makefiles, but did not find any obvious place
> > to put the locking.  I probably just did not look hard enough.)
> 
> The ports system is to "install" a new port.  It won't be easy to accomplish
> what you suggest.  For example, dependencies are checked one at a time.
> So, even if you want to run multiple processes on LIB_DEPENDS, there is no
> easy way to control CPU load.

What I suggested should not present any major difficulties.  I did not
propose automatic parallelization, or anything that needs to control CPU
load in way.

What I wanted is just support for manual execution of two parallel port
builds.  E.g. I want to be able to write
'cd /usr/ports/x11/gnome2 ; make install' in one virtual terminal and then,
while the first command is running and installing all kinds of dependent
ports, to able to switch to another terminal and write
'cd /usr/ports/x11/kde3 ; make install' without having to worry about the
fact that some ports (like x11/xorg-libraries) will be needed by both
builds.  (Assume that I have already done a 'make config-recursive' for both
gnome2 and kde3.)

Currently doing the above will not work reliably.
All that should be needed to make it work reliably is to add some kind
of locking so that the ports system will not try to build a port if a build
of that particular port is already in progress. 
(I suspect that using lockf(1) at appropriate places in
/usr/ports/Mk/bsd.port.mk  would be sufficient, but there are almost
certainly a bunch of details and corner-cases that need to be considered.)



> 
> It is a better idea for other "ports UPGRADE" utilities to take care of your
> suggestions.  Indeed, I have been developing such utility myself.  If you
> want to try, I can give out for testing.  There are 2 known issues with my
> tool, yet: 1. no good way to run 'make config', yet, and 2. even if
> less LIB_DEPENDS are required due to less selected OPTIONS, my tool does
> not fully eliminate these dependencies.
> 
> Hiro

-- 
<Insert your favourite quote here.>
Erik Trulsson
ertr1013 at student.uu.se