Call for testers for yet another ports upgrade program, ports+

Yoshihiro Ota ota at
Fri Jul 27 03:36:46 UTC 2007

On Thu, 26 Jul 2007 02:17:50 -0700
Garrett Cooper <youshi10 at> wrote:

> Yoshihiro Ota wrote:
> > To Whom Slowness of Portupgrade Concerns a Lot:
> >
> > As I got tired of long waiting of portupgrade trying to resolve dependencies, I came up with
> > yet another tool for upgrading FreeBSD ports system.  Unlink other tools, it tries to maximize
> > existing resource to maximize its performance.  This program attempts to wrap around with
> > another 'make' and expand use of FreeBSD ports system.  The heart of ports+ is parsing INDEX
> > and +CONTENTS files.  The rest is handed to GNU make.  I think it comes to a point where I seek
> > wider audiences to test with it.
> >
> Hiro,
> You're correct when you say that reading INDEX* does take a long time
> for portupgrade, but the problem is partly with how portupgrade formats
> its version of the INDEX database in BDB format, as well as how it
> doesn't buffer up proper information (pkg database information for
> instance), and guesses at port dependencies (origins, deporigins, etc).

Portupgrade is not only slow reading INDEX file but also on dependency resolving and updating +CONTENTS files, too.  I think portmaster is also one tries to read and do the same things but with shell script.  I personally didn't have good luck with portmaster and haven't really used to evaluate.  However, "portmaster -a -n" wasn't not fast, neither.  By the way, it builds ports in background, doesn't it?

> 1. Can your solution be made bsdmake and bsd awk compatible?

I am not as familiar in newer BSD awk and make extensions.  I can point out what kinds of GNU extensions are being used so that someone can tell it is also available in BSD make/awk.

For awk, this feature is required, quoted from "man gawk".  I am not sure if BSD awk implements this feature.

       The book indicates that command line variable assignment  happens  when
       awk  would  otherwise  open  the argument as a file, which is after the
       BEGIN block is executed.  However,  in  earlier  implementations,  when
       such an assignment appeared before any file names, the assignment would
       happen before the BEGIN block was run.  Applications came to depend  on
       this  "feature."   When awk was changed to match its documentation, the
       -v option for assigning variables before program execution was added to
       accommodate  applications  that  depended upon the old behavior.  (This
       feature was agreed upon by both  the  Bell  Laboratories  and  the  GNU

For make, 1.1 and 1.2 below are in the same topic.  I use GNU extension which keeps generating makefiles until dependency rule saticifies and use it part of the make rule.  GNU extension reloads newer makefiles and reconstruct dependencies.  I am not sure if it is possible in BSD make; however, this is not a heart of this program.  We can work around with bootstrap script or something.
1.3 This is recursive expansion of variables.  If you take a look, every port needs upgrade uses $(UPGRADE).  $(UPGRADE) is expanded multiple times using the target variable, $@, such that it backups up correct old package names and so on.  This is very important and unless this works in BSD make, my tool really needed full redesign from scratch.
1.4 is just how GNU make does shell invocations.  It is available in most of Make tools.
1.5 is something I am not sure which makes support these.  I don't think GUN make recommends using special characters as variable names.  However, when I tried, it was too difficult to normalize to alphabets and underscore only.  If we cannot use variable like PORTS+IGNORE as a variable name, it will be very difficult and personally I don't even think about trying that myself.  If you take a look at compat.gmk which ports+ generates as a part of its build rule, you can find something like "COMPAT_$(PKG_DIR)Hermes-1.3.3_2 = cp -f /usr/local/lib/ /usr/local/lib/compat/pkg&&" being variable.

1.1 How makefiles get remade:

1.2 Including Other Makefiles:

1.3 Computed Variable Names or recursive expandsion of variables:

1.4 The `shell' Function

1.5 Special characters in variable names such as dot, ., comma, ,, slash, /, and so on.

> 2. Does your solution account for cases when you're trying to install
> package a, which depends on package b, but because you built package b
> while trying to build a, and are at an intermediate package c (between a
> and b), the dependencies for a are only partially complete. Thus when
> you try to install direct or indirect dependences, the install fails?

Some of answers are explain in the GNUmakefile.  Anyway, ports+ categorizes all ports into three, NEW as ones not installed, UPGRADE as installed and newer version is available, NOOP as installed an no newer version is available. I only started with upgrading only and haven't really though thought installing new ones.  I am not ready to talk about installing new ports via ports+.   Short answer for upgrade is, "Yes, it does."  Here are some examples.  Let's say 'a' depends on 'b'.  Then, it builds 'b' first and than 'a'.  If new version of 'b' has new requirement of 'c'.  Making 'a' will result installing 'c' and than upgrading 'b.'  If 'e' requires 'f', 'g', and 'h' and these are independent from each other, upgrading 'e' with -j 3 will start upgrading 'f', 'g', and 'h' at the same time and once all three are upgraded, it will start on 'e'.  

> 3. How is this different from pkg_version combined with similar scripts?

I wasn't aware of pkg_version, but it looks like it only tells if port is up to date or not.  It doesn't tell anything about dependencies among ports, does it?  If so, it doesn't provide enough information to upgrade.

The difference from other two is the following.  Portupgrade and portmaster "reads" what's in ports, INDEX, and what's installed, +CONTENTS, "solves" dependencies itself, and "manages" upgrading port itself.  Instead, ports+ "converts" INDEX and +CONTENTS to (GNU) Makefile and let the dependency expert solve the rest.


> -Garrett

More information about the freebsd-ports mailing list