Parallel Builds

[Doug Barton]

Benjamin Lutz wrote:
> Hello,
> 
> Since Multi-core processors are becoming popular (or, more 
> egocentrically, since I've acquired one), I've become interested in 
> parallel compilation. Unfortunately, it seems that parallel builds of 
> any kind are completely unsupported by the ports framework at the 
> moment.

Not for lack of interest, but for lack of person hours to do the work. 
As you've already ascertained, the amount of work required is 
substantial, and would amount to a constantly moving target, since new 
ports are always being added, and "vendor code" is always changing.

>  o Leave the ports framework as it is, and implement support for
>    parallel building in add-on tool, eg., portupgrade. The tool would
>    support automatic parallelism ("portupgrade -a" would automatically
>    build ports in parallel where possible), or having several
>    user-created instances running at the same time. I'll call this
>    "tool-based macro-parallelism".

I specifically designed portmaster to be able to do this. Separate 
instances do not share anything in common, other than the ports tree 
and package database. Assuming that you were updating two (or more) 
ports that you were positive did not share any dependencies in common, 
you could run as many portmaster instances as you wanted to, and not 
have any problems.

Where this gets really sticky is in (for example) the -a case. If you 
run 'portmaster -l' on a typical system, you'll see that there are 
some root ports (no dependencies, not depended on), some leaf ports 
(not depended on), but most of the installed ports are in the 
"middle," where they are depended on by something else, and most of 
those have dependencies of their own. It would be a SMOP to have 
portmaster track the ports that are "safe" to upgrade in parallel. 
Then of course you have to keep track of your child processes, create 
a queue for what ports to run next, etc. etc. It's all doable of 
course, but I think the return on investment for this project would be 
very small. Patches are always welcome however. :)

>    Disadvantages:
>    - Moderately difficult to implement.

*cough*

> Locking, build failures and interruptions would have to be taken
> care of. I don't see problems that can't be solved though.

Yeah, like I said, it's all about having the time to do it.

>    - No speed gain when updating single large ports, eg. gcc. (To be
>      fair, it must be said that some of the large ports, eg.
>      OpenOffice.org, don't support micro-parallelism either. Macro-
>      parallelism would at least allow the otherwise unused CPUs to
>      do something sometimes.)

My gut feeling is that the cases where the gains outweigh the effort 
required are very few, and far between. Maybe someday when 64-way 
desktops are the norm, sure, but not now. :) I would of course not 
ever discourage someone who wanted to work on this, it's just not on 
my TODO list atm.

hth,

Doug

-- 

     This .signature sanitized for your protection