Keeping Ports and Packages Synchronized

[Mark Linimon]

On Thu, Jul 26, 2007 at 04:27:37PM -0700, Kurt Abahar wrote:
> However, I don't know how to get a hold of this "lag
> time." Is it a few days, a few weeks or ... ?

You can get an _idea_ of the degree of the lag via the following URL:

  http://pointyhat.freebsd.org/errorlogs/packagestats.html

The "cvs date" column shows you the date of the CVS update for each
individual build environment (which is architecture * OS branch), _if_
the standard build scripts were invoked which automatically create this
file.  However, if for some reason the CVS update had to be done manually
in the first place, or an update to certain ports was done after the
initial CVS build, then that file might not be correct.  (Reasons for
having to do this might involve trying to update a single failing port
with many dependencies, or the CVS update having caught the tree in a
state where INDEX was not consistent, and so forth.)

Depending on the load on the port build cluster machines, amd64 and i386
builds can take a few days to longer.  The sparc64 package builds take
weeks (we have far fewer sparc64 machines, and they are much slower).  You
can safely treat the ia64 package builds as a mere sanity-test of the ia64
src tree; since they take over a month, they are useless for actual packages.

The other columns will give you pointers to the latest INDEX that was
created from each checked-out CVS tree; the number of ports marked IGNORE
for some reason ("skipped"); and other things.  (See the text at the
bottom of that page for a fuller explanation).

But to answer your original question: the only way that we could keep
packages 100% in sync with the source would be to have two completely
separate versions of the source; one that was "internal" and one that
was released to the public.  But which timeframe do you use?  That for
amd64/i386, or the longer ones?

Also, note that these runs _overlap_.  As one of the (e.g.) amd64 builds
winds down to the last few long-running packages, we start another one for
separate branch.

Between the fact of this, and the fact that the Ports Collection is an
infinitely moving target, there is _no_ time 'T' where the packages for
-stable and -current are up-to-date.  The only case this is guaranteed
to happen is when we tag the ports tree for each release and then build
packages based upon it.  Even with that, we do each package set and then
have to further manually update ports that have security updates, and
re-package them.  We _hope_ to not make errors in that work.

To summarize: the problem is that there are too many moving parts that
all move simultaneously.  The only way that we can guarantee that the
packages match your ports tree is at release time, and only then with
a great deal of QA.

(Note that I have skipped, in this discussion, packages that we can not
make available for license reasons, and the packages that depend upon
them; and packages that are currently not being made correctly*, and
the packages that depend on _them_.)

Now, in _general_ the amd64/i386 packages will be "fairly close", for
some value of "fairly close".  This becomes untrue when, e.g., a commit
is done to the ports infrastructure, or a port that affects a large
number of packages (gettext, perl5.8, python, autotools, autoconf, and
so on.)

But the general-case problem is very, very, hard.

mcl

* either for temporary reasons, reasons that they don't yet build on that
architecture and/or OS release and/or gcc release, and other things