Utility for safe updating of ports in base system

Michel Talon talon at lpthe.jussieu.fr
Thu Mar 20 02:44:02 PDT 2008

On Thu, Mar 20, 2008 at 01:05:27AM -0700, Doug Barton wrote:
> On Thu, 20 Mar 2008, Michel Talon wrote:
> >Doug Barton wrote:
> >i would venture to say that such an utility
> >should be able to upgrade things based of *binary* packages, and
> >consequently that portmaster is not a suitable candidate.
> That ability is not included in the current requirements document, and was 
> not specifically mentioned the last time we had the discussion on the 
> list. If the portmgr folks intend that to be a requirement, the current 
> ideas list entry should be amended.

Indeed you are right, but i think this should be a requirement for the
reasons discussed below, and is implicit in the fact that the projects
refers to a "portupgrade written in C", and portupgrade has such a

> >For example
> >pkg_add installs a binary package, if you want to compile and install
> >you run "make  all install clean" in the ports tree.
> Um, you lost me there.

This is simple, all pkg_* tools operate on binary packages at present,
it would be strange that a newcomer, pkg_upgrade has no way to operate
on binary packages.

> >One of the
> >requirements of an upgrade system is predictability, this can only
> >be achieved by using binary packages.
> You gain a certain amount of flexibility with packages, at the expense of 
> being able to customize things. As long as the user understands that, then 
> it's fine.

I agree that the possibility of compiling from source brings ability to
costumize things. However this very ability ruins all hope of having a
smooth upgrade system. Due to customization, as soon as you have many
ports installed, there is a combinatorial explosion of possibilities and
nobody can test that the upgrade process works. OpenBSD people have
been converted to this idea, and now push for an upgrade from binary
packages exclusively.

So i think that the user
should have two possibilities:
- either he wants to customize, and he is on his own. He will need to
  compile his software, and in this case portmaster is the perfect tool
for managing his upgrades. Since the compilation will take most of the
time, it is not relevant to consider performance questions on the
portmaster side.
- or he wants to use a tried and true upgrade path, he doesn't want to
  spend time compiling, he wants speed. Basically he wants the Debian
or Ubuntu experience. In this case, using prebuilt binary packages is
the only reasonable way of achieving the result. I agree that due to
licence peculiarities (think e.g. Java) there are a small number of
ports which will need compilation, so the experience cannot be perfect.

> >Another requirement, in my opinion,
> >is speed, and the lack of speed, which is completely hidden when you
> >compile your packages will be immediately apparent if you try to use
> >packages. Indeed portupgrade has options -P and -PP to work with
> >packages which could serve as a prototype for a "pkg_upgrade" written
> >in C, except that they work poorly, and in particular run slowly.
> Where do you think the slowness is?

Several causes of slowness have already been identified. Parsing
/var/db/pkg is slow, this is why the idea of using a database has been
advocated. Much worse, running make in a port directory, even only to
get the value of a variable is slow. Programs like portupgrade do such
things repeatedly without caching the results in memory and incur
large time penalties. Similarly for each package he wants to download,
portupgrade opens an ftp connection and retreives it independently, etc.
Obviously no effort at all has been spent so that things are fast.

> >One of its features, that i consider very important for correct
> >operation, is that it computes the list of all packages to be deleted
> >and all packages to be installed and asks the user if he agrees before
> >doing anything.
> Why do you consider this an important feature? (I'm not disagreeing, just 
> curious about your thought process here.)

Because you can see that something is going to break in advance and
renounce to the upgrade. This occurred several times for me on Debian.
Usually this is because some package has a security problem, and this is
solved waiting one or two days.

> >It fetches all necessary packages before installing or
> >deleting anything.
> That seems sensible, thanks for mentioning that bit.
> >Hence you can be sure that the upgrade process will
> >not end in a mess if something crashes in the middle, like it is the
> >case with all present standard FreeBSD upgraders.
> Not sure if this helps the situation you're referring to or not, but 
> portmaster will by default make a backup package of each port that it 
> updates, so if something dies in the middle you could back out of it by 
> hand if you need to.

Yes, so does portupgrade, but it deletes the backup as soon as the
installation of the new port succeeds. This means that in the event of
a crash in the middle you remain with a half upgraded system, and no
way to backup completely to the previous working state. In my
experience, such half upgraded systems sometimes work OK, and sometimes 
half work.

> Now all that said, I'd love to see us move to a much more robust package 
> management system, or even just a better interface to the one we have. The 
> problem is that I don't have the time to do that as a volunteer project, 
> and I don't think anyone else does either. :)

Probably the present package management system doesn't keep enough state
on each installed package so that an upgrade system works reliably. For
example, referring to the possibilities to tweak options, i think that
keeping a copy of the options used for compilation of the port in the
package could help, or at least a flag indicating if the package is
basically standard or has been tweaked. Similarly keeping track of
wether a package has been installed by order of the end user or as a
dependency could help to unclutter after an upgrade. On the other hand,
basic libraries, which, when upgraded, require to upgrade a lot of ports
should have a better way to require that, rather that asking the end
user to read UPGRADING. Finally both the speed questions and the
necessity of keeping more state point to the idea of using a database
instead of abusing the filesystem for doing a database job. But this is
precisely one of the Soc projects.


Michel TALON

More information about the freebsd-ports mailing list