compiling ports with more than one job

[RW]

On Mon, 5 Mar 2007 22:45:53 -0000
"Thomas Sparrevohn" <Thomas.Sparrevohn at btinternet.com> wrote:

> There really two answers possible here - 
> 
> 1) Let's call it one depth e.g. make -j - Which works with some not
> all ports - Nice when it works and I guess ports/Mk could hold a flag
> 2) Let's call it width - e.g. the ability to compile packages at the
> same time given that all dependencies has been resolved
> 3) combination of 1 and 2 
> 
> In practical terms option 2 is much more attractive as it is possible
> to determine that just from the INDEX file and the installed ports. 

You would probably have to do it like portmanger does, the INDEX file
isn't accurate. 

> However due to the way compilation options are treated e.g. I am not
> sure that it is completely safe - I will require some locking 
> during the make (de/re)install phase but possible to handle - It
> would still cut portupgrade significantly 

It couldn't be done easily in the ports system itself, a new tool would
be needed. However I think it can be done without any locking at all. 

The way I would do it is have a single master process that works
through the packages in dependency order. For each package it does a
"make configure"and then spawns off the build in a separate
process. Once N builds are in progress, the master process sits there
waiting for processes to complete, serializes the installs and kicks
off new build processes.

The beauty of this is that anything in the least bit critical is done
in the single master process, but most of the work is done in
the build processes, so the master wont be a bottleneck. There's a bit
more to it, for example there would have to be special handling for
packages that appear to have missing dependencies, consequently  a
different logic would be needed for new installs.



> With 15-16000+ ports I think that 1 and 3 are unpractical - however

I don't think it is, it could be handled the way amd64 ports are/were
handled, by asking people to try them and report back.