Request for Features: Ports Re-engineering

[Alejandro Pulver]

On Mon, 17 Dec 2007 12:07:35 -0600
linimon at lonesome.com (Mark Linimon) wrote:

> On Mon, Dec 17, 2007 at 12:59:54PM -0300, Alejandro Pulver wrote:
> > if not already done, AFAIK tinderbox has an option to run parallel
> > builds, but don't know about pointyhat) using multiple processors.
> 
> pointyhat is actually the dispatch machine.  It does not actually
> build packages.
> 

Of course, I should've said it another way, but I was referring to the
build scripts that run in the build clusters.

> The current machine total:
> 
> amd64:    3 (2.2GHz)
> i386:     40 (400MHz - 1.8GHz, depending on machine)
> sparc64:  7 (440MHz)
> 

Interesting, I only knew about the 3 new amd64s (for which you posted a
mail) but didn't know the details of the rest.

I remember some of my ports failed on ia64 (some time ago), what
happened to it/them?

> Several of these systems (in particular the amd64s) have multiple CPUs in them.
> 
> In any case, each machine is given several packages to work on simultaneously,
> depending on its capacity, each in its own fully clean jail.  Prerequisites
> are loaded via the previously built packages; if they are not available,
> they are first built, then cached.
> 
> The way the current code works is documented in the following article:
> 
> http://www.freebsd.org/doc/en_US.ISO8859-1/articles/portbuild/index.html
> 
> Although the documentation is terse, it is believed to be generally
> correct.
> 

Will read it, didn't know about it.

> In general, we use -incremental for the builds, unless there has been
> a large change in the src tree, or it's release.  -incremental will only
> build packages for ports whose metadata (as determined by INDEX) has
> changed since the last run.  This holds the build time for each package
> set down to a few days (amd64/i386) or a couple of weeks (sparc64),
> instead of around a week, or a couple of months, respectively.
> 
> Remember that we have builds for 5.x, 6.x, 7.x, and 8.x all running
> at various times (some things can be overlapped without adding too much
> delay).  The machines are almost never idle.
> 
> I strongly suggest that before designing a new system, some careful
> attention be paid to how extensive and optimized the current system already
> is.  I don't think the people following this thread are going to get the
> right perspective on the scope of this work, otherwise.
> 

To clarify: this rewrite of the current ports system does not include
(at least for the moment) a replacement scripts/program for the build
machines. I didn't say the build scripts could be optimized, I just
said that if the system is improved in general less time would be lost
in INDEX generation, dependency gathering (well, AFAIK portbuild takes
it from INDEX so it won't be different), package registration/removal,
etc. a little speed would be gained.

Best Regards,
Ale
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-ports/attachments/20071218/30e793c6/signature.pgp