Limitations of Ports System

[Aryeh M. Friedman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yoshihiro Ota wrote:
> On Thu, 13 Dec 2007 21:58:57 +0100 Erik Trulsson
> <ertr1013 at student.uu.se> wrote:
>
>> One shortcoming is the lack of locking making parallell builds a
>> bit unsafe. If you try to build both port A and port B at the
>> same time, and both A and B depends (directly or indirectly) on
>> port C which is not installed, then you can esily end up having
>> two processes both trying to build C at the same time.  This
>> usually confuses both builds very badly making them fail.
>>
>> I also don't think there is any locking on /var/db/pkg making
>> possibly somewhat unsafe trying to register the installation of
>> two ports/packages at the same time.  I have never noticed any
>> actual problems with this though.
>>
>>
>> Some sort of locking, making parallel builds safe, should be
>> possible to add to the ports system without doing any sweeping
>> changes. (I did look briefly at the makefiles, but did not find
>> any obvious place to put the locking.  I probably just did not
>> look hard enough.)
>
> The ports system is to "install" a new port.  It won't be easy to
> accomplish what you suggest.  For example, dependencies are checked
> one at a time. So, even if you want to run multiple processes on
> LIB_DEPENDS, there is no easy way to control CPU load.
>
> It is a better idea for other "ports UPGRADE" utilities to take
> care of your suggestions.  Indeed, I have been developing such
> utility myself.  If you want to try, I can give out for testing.
> There are 2 known issues with my tool, yet: 1. no good way to run
> 'make config', yet, and 2. even if less LIB_DEPENDS are required
> due to less selected OPTIONS, my tool does not fully eliminate
> these dependencies.
>

Your correct that there are 2 seperate issues at play here but there
is a common solution (and to be honest I have yet to see any
feature/issue discussed in any of the re-engineering threads that
doesn't at least become more manageable under this general design
concept I am working under).... I hate to keep referring to Miller97
but I think it highlights (directly or indirectly) every single issue
that has been discussed while a little off topic (and slightly self
serving) there is a good explanation of  the general idea behind what
I have in  mind in the  cook tutorial (I am the author thus it is
self-serving)
http://miller.emu.id.au/pmiller/software/cook/cook-2.30.tut.pdf..

In the the specific case of parallel builds once we pre-scan the DAG
it is trivial to do a *FULL* DFS on it and just say for any time two
ports are not in the same DFS generated subtree in respect to some
root target (can be recursive) they can be build in parallel.  Locking
is also trivial now that the decision on ordering is made by the ports
system and not indivual ports makefiles. (the indivual make files are
still needed to build the port but should not and by definition can
not contain knowledge about their depends).

Side note the more we discuss this the more obvious it becomes to me
it has to be in some OO lang and since C++ is the only one in the base
system it kind of forces C++ to be the implementation lang.

- --
Aryeh M. Friedman
FloSoft Systems
http://www.flosoft-systems.com
Developer, not business, friendly
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHY1hWzIOMjAek4JIRArNwAJwMEsZVVMTnl3F4T96BfWGY/PHy2ACaA/RZ
NGtCCzJp3z90MwP/UWGrp5o=
=tTt4
-----END PGP SIGNATURE-----