parallel builds revisited

[youshi10 at u.washington.edu]

On Thu, 12 Apr 2007, Benjamin Lutz wrote:

> On Thursday 12 April 2007 18:32, [LoN]Kamikaze wrote:
>> Robert Noland wrote:
>>> Have any of you looked at sysutils/bsdadminscripts, it's buildflags
>>> options allow for parallel builds as well as ccache / distcc use.
>>> I have a reasonable list of ports that must have some or all of
>>> these options disabled as well.
>>>
>>> robert.
>>
>> I happen to be the author of that and I maintain a list of ports that
>> cause trouble in a German Wiki:
>> http://wiki.bsdforen.de/Distcc#FreeBSD_Ports
>>
>> These are all the ports that ever made trouble when I tried to build
>> them with 'make -j'. I have around 500 Ports installed, so I think
>> those aren't really many.
>
> I've looked at your patches and programs, and I'm starting to have a
> fairly clear idea of solution should look like. I would like to see:


Benjamin,
       A few things:

> * Integration into the existing ports framework. No new scripts or files
>  should be required. The whitelist file needs to go.

Yes. I'm thinking that a knob

> * The whitelist should remain, even in the final version (for when users
>  feel they know better than the port maintainers whether the software
>  supports parallelism or not). I will turn it into a make variable
>  though.

My personal opinion is that the document should be a reference item that everyone can modify, such that devs / maintainers know that building with parallel options is possible or impossible with multiple jobs.

> * Configuring the number of jobs should be automatic (but overridable),
>  based on kern.smp.cpus.

Yes.

> * If a port build fails, and parallelism is enabled, print a message
>  telling the user to repeat the build in single-process mode before
>  reporting the error, *unless* ALLOW_MULTIPLE_MAKE_JOBS (or something
>  like that) is set in the ports makefile, which is interpreted as
>  this feature being officially supported by the port maintainer. So
>  basically, if you make use of the whitelist feature, you do so at
>  your own risk.

My personal opinion is that it, if it were to fail, then the user should have the option of specifying a flag or knob which would automatically retry the build if requested and display summary info about the build at the end (what failed to build, what didn't), so the user can email the maintainer with more info or get support if necessary.

> * There is no need to put any references to distcc or ccache into the
>  code. Once -j works, users can get distcc or ccache support simply by
>  setting CC apropriately. I think.

Yes, definitely.

> Btw, do you think it's possible that a port can only be built with, n
> parallel make jobs, but will fail with n+1?

I highly doubt it. Unless one portion fails (or in the case of distcc some information isn't communicated across a cluster doesn't make it to the master and back to the slave(s)) and there's a dependency, stuff should just go a bit slower. That's a hunch though, based on my personal experience with the number of make jobs I've scheduled on my Gentoo box in the past with gmake.

-Garrett