ports and dependency hell

[Julian Elischer]

This is a serious post  on a serious issue that ports framework people 
seem unaware of.

It' getting too easy to get into dependency hell here (I've spent the 
last week fighting this)

In  a production system we have to choose packages from different eras.

This is because on an average product different teams are responsible 
for different parts of the product, and they all have their own 
requirements. Not to mention the stuff that comes in from third party 
vendors which "must use version X of bar and Version Z of foo". This 
is something that is a fact of life in commercial vendors. Ports needs 
to support it, and fast because currently ports is a reason to switch 
to Linux. (ammunition for the Linux fanboys). We are only staying for 
the ZFS support but that reason will evaporate soon.

We may need node.js 6.95 and a particular version of an apache mod for 
example, and a specific version off npm, which all only appeared in 
the ports tree at different times, so either we completely ditch the 
ports tree all together and import buildroot2 , which allows every 
package to have its own revision (but is Linux centric), or we need to 
generate frankenports trees. My curent iteration has 359 different 
packages, so you an imagine the hilarity  if I need to slide 20 of 
those back or forth, all independently.

There doesn't seem to be any understanding of this fact from the ports 
framework, and it means one has to keep one's own ports tree in source 
control, as a branch off the FreeBSD one. (maybe I should look at 
pkgsrc)..

the problem is that the internal APIs of ports are changing too much.

If you are going to change the API, then you need to be able to 
declare the version separately, maybe in a version/distinfo file that 
can be pulled in separately at a different rev, rather than having it 
built into the main Makefile of each port. Maybe the Makefile 
specifies a revision range it can be used with, but that would make a 
huge improvement right there.

You can not just slide one port up by 3 months, and another down by 3 
months to get the revs one needs because the damned Mk files have 
changed. If I coudl leave the bulk of the Makefile alone and just 
slide the 'distingo/rev' file, (or be able to select a rev from one 
htat gives many alternatives, that woudl be "wonderful".

Please, ports framework people,  think about how this can be done, If 
I have to edit a file, the game is lost.

Can we please get some understanding from ports people that they are 
making things REALLY HARD on vendors and it really strengthens the 
hands of  the "ditch FreeBSD and go to Linux" crowd when I need to 
spend a whole week trying to integrate in a set of 5 new ports into 
the product.

The call "It just works under linux, select the versions you want of 
each package and type make" is often heard around the company. And 
management is not totally deaf.

If you want to see how its' done better (still not perfect), go build 
a busybox system. you can effectively select any version of any tool 
and they all communicate via the pkgconf mechanism and you get the 
result you want.(mostly). And the API is stable.

On the pkg side of things we need the ability for pkg to say "yeah I 
know I'm looking for foo-1.2.3.txz as a perfect match, but I've been 
given some slack on the third digit and I can see a foo-1.2.1.txz, so 
I'll install that instead".

Otherwise we just have to spend WAY too much time generating dozens of 
"matching sets" of packages , that must be kept around forever if just 
one machine shipped with that set, not to mention the fact that making 
the matching set is often a real task.

The way to get around the problem above CAN be (not always) to install 
foo.1.2.1 first and THEN install the package you actually want, and 
pkg will accept it. The problem comes when pkg needs to install a 
dependency itself. Then it becomes "super picky", when there is 
actually a range of package revisions that would do. Instead of 
letting pkg install what it needs, we need to manually set up scripts 
to install the dependencies. so that all the work done by pkg is wasted.


Please think about these two issues..


Julian