Deterministic package building with ports

[Peter Schuller]

Hello,

> One thing I'm confused about here, what are your goals? You don't
> mention why you need to build the packages. The answer to this might
> inform the rest of the conversation.

At the very highest level, the goal is to keep a system, or a set of
system (large or small) up-to-date in an efficient manner, while
minimizing downtime. I want to build packages separately not just to
minimize the time window, but also to ensure that the full set of
packages can successfully be built - avoiding a half-upgraded system
that has issues to be resolved.

The system(s) is/are assumed to need custom port options; thus use of
bulk built packages is not an option.

> > For the past half year or
> > so I have ended up using a hacked together shell script which does
> > approximately what I need, which is:
> > 
> >   To build deterministically as a function of
> >     (a) base system
> >     (b) /etc/make.conf
> >     (c) /usr/local/etc/ports.conf (sysutils/portconf)
> 
> These are givens, but it never hurts to mention requirements explicitly.

My intent was also to convey that I want it to be a function of *only*
the list of things I provided, rather than an unknown set of factors.

> >     (d) a list of specifically desired origins/packages which is manually
> >         mainteined (NOT a complete list of dependencies, this is critical)
> 
> Why is not maintaining the list of dependencies critical? I'm guessing
> the answer is related to the dependencies being fluid over time.

Because it becomes a huge pain when your complete package list is 1000
packages. What always ended up happening back when I tried keeping
machines up to date by portupgrade/portmanager, was that every now and
then I would just get fed up with all the built-up issues and
re-install everything from scratch.

The actual real-life goals are often very easily expressed in terms of
a pretty small set of packages that one is actively interested in. Any
manual maintenance of a very very significantly longer list of
dependencies is just manual labor that you want to automate. I view it
as the package management equivalent of maintaining a set of patches
against a source tree instead of just forking it. I have something
that I can realistically maintain manually which expresses exactly my
intent, nothing more, nothing less.

When writing pkgmanager (pkgsrc only) this was a major goal. You
maintained your want list, and your pkgsrc tree, and you just told it
"sync". It then did whatever it needed to do to produce a system which
is identical to what you would have gotten by taking your list of
desired packages and "bmake install":ing them on a clean system. It
was all a function of a defined set of things, and the process was
100% restartable because all necessary discovery happened based on
persistent information on disk. In short, you told it what you wanted,
rather than what to do.

Of course, this didn't work perfectly - mostly due to non-determinism.

> >     (e) the ports tree

[keeping the above to keep the quoted list complete]

> > Is it the intent of the design of ports that the behavior of packages
> > be implicitly dependent on the state of installed packages?
> >
> > If the answer is yes, can someone suggest a practically viable that
> > consistently works, method of building binary packages under the above
> > conditions?
> 
> Yes, this is a feature. You could solve the problem you describe above
> by building the X stuff first.

The problem is that I do not want to care about that. I know that "I
want aumix" and that "I want firefox". I do not want to discover by a
long period of trial and error that I need to fudge my tools to
install aumix prior to firefox, or the other way around.

We have even more trikier issues like "I want subversion" and "I want
apache", and having to specify which apr has to be used. I see no
general and simple way of implementing detection of such conflicts in
a package manager built on top of ports, without modifying ports to
accomodate.

If being impacted by installed packages is considered a feature, it
may be that my preferences/goals are just fundamentally incompatible
with that of ports.

Would there be interest/opinions on having ports support a mode of
operation specifically targetted at automated tools and determinism? I
suspect having a separate mode will just lead to too much complexity
and breakage.

> > (I know of portupgrade/portmaster of course. I have never found a tool
> > that both (1) does what I want 
> 
> What you want is beyond the scope of current tools. I have a proposal
> for adding this functionality to portmaster but it is a considerable
> amount of work and therefore I was hoping to get some funding to support
> it.

Do you think it is appropriate to solve this elaborately in tools, as
opposed to making changes to the ports infrastructure to facilitate
these things? Am I barking up the wrong tree, hoping to see the
latter?

I have a genuine interest in having something which works and is
reasonable simple to use for an administrator, and am willing to
spend time on it. But not if the work is going to be linear with
respect to various quirks and oddities of the underlying system. In my
personal ideal world, the fact that the installed set of packages
affects the dependency decision of a port would be a bug. I take it
from the above that this is not consistent with the generally accepted
view.

I have planned for a long time to re-write pkgmanager and support both
ports and pkgsrc; but fundamental issues such as determinism are
putting me off (other than lazyness) because I know that it will never
really work as well as I would want it to in order to be truly useful
and more effective than the hacks I am currently using as a poor man's
alternative.

> > and (2) actually works consistently.
> 
> If you wanted to list the ways that portmaster does not work
> consistently in a different thread I'd be interested.

I want to clarify right here for the record that I am not accusing
portmaster of not working consistently. However, it did fall into (1)
- that is, that it does not do what I personally want. portupgrade
however does fall into both categories in my book.

I'd be very very interesting in discussing package management and tool
issues if you are willing. However, I'm not sure whether my personal
goals are sufficiently in synch with yours or others for it to be very
fruitful. I could try to recall past experience and issues I have run
into administratively and when developing pkgmanager and try to state
more clearly what my biased view is, with respect to what I would like
tools to do, and how I think ports could facilitate those tools. Would
freebsd-ports@ be a suitable forum for that?

-- 
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <peter.schuller at infidyne.com>'
Key retrieval: Send an E-Mail to getpgpkey at scode.org
E-Mail: peter.schuller at infidyne.com Web: http://www.scode.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-ports/attachments/20081203/7c764a2f/attachment.pgp