The cost of a source based package system

[Erwin Lansing]

On Thu, Sep 08, 2011 at 04:45:30PM -0700, Xin LI wrote:
> 
> Both portmaster and portupgrade have 'package' mode, which uses
> packages when available.  If one can live with default optimization
> (which is usually good anyways) and if most times the default options
> would satisfy his/her need, or if the port doesn't provide any
> options, binary packages would save a lot of time.
> 
> The real problem for FreeBSD's packaging system is, in my opinion, we
> do not maintain branches and ports tree is a fast moving target,
> making it impractical to build packages and push to mirrors.

Absolotely right.  There is also so ongoing work to move away from the
current model.  We already are using a consistent model for from which
src branch we build packages on.  Previously, this was RELENG_X on a
given day, where that day was defined by "whenever the portmgr running
that architecture has time to do so".  This is now oldest supported
minor release within each major branch.  More on:
http://www.freebsd.org/portmgr/policies_eol.html

We also need to do this for the ports tree.  The best way to do so is to
provide consistent package sets which are not overwritten on the mirrors
when a new package set becomes available, but let the user move between
sets manually.  This requires some large changes, both to the code and
infrastructure which we are working on.  The code needs to be able to
identify which package set is installed and an ability to move to a new
set.  The PKGNG project will provide us that.  Also, the current mirror
infrastructure cannot support the extra amount of data needed.
Currently, a full set of packages for one architecture is about 30G.
Just supporting the Tier-1 architectures on 2-3 live branches add up,
especially if we also need to keep multipe full sets around.

There's a short presentation from BSDCan with some bullet points:
http://people.freebsd.org/~erwin/presentations/20110512-BSDCan-packages-summary.pdf
This is also a topic we'll talk more about at EuroBSDCon next month.
> 
> My $0.02: It might be worthy to experiment a branched development
> model and only pull up changes at a much lower pace to branch (e.g.
> create a branch near a release and drop the branch after a few weeks
> once a new one is created, and only pullup changes when there is need,
> like because security vulnerability or serious reliability/performance
> issue), it would be easier to produce binary package and sync them
> across mirrors.
> 
I don't think brances is the right solution here.  Talking to the pkgsrc
people, they use quite a lot of time on pullups and we have almost 3
times as many packages.  This would not scale to ports.  Unfortunately,
as I do like the model in principle.  I think we can come near though
wil clearly defined EOL/EOS lifetimes for the package sets (see slides).

Erwin

-- 
Erwin Lansing                                   http://droso.org
Prediction is very difficult
especially about the future                    erwin at FreeBSD.org