ports and dependency hell

[Steve Wills]

Hi Julian,

On 02/07/2017 13:03, Julian Elischer wrote:
> This is a serious post  on a serious issue that ports framework people
> seem unaware of.

To be honest, it's kind of a confusing post, at least to me.

> It' getting too easy to get into dependency hell here (I've spent the
> last week fighting this)
> 
> In  a production system we have to choose packages from different eras.
> 
> This is because on an average product different teams are responsible
> for different parts of the product, and they all have their own
> requirements. Not to mention the stuff that comes in from third party
> vendors which "must use version X of bar and Version Z of foo". This is
> something that is a fact of life in commercial vendors. Ports needs to
> support it, and fast because currently ports is a reason to switch to
> Linux. (ammunition for the Linux fanboys). We are only staying for the
> ZFS support but that reason will evaporate soon.
> 
> We may need node.js 6.95 and a particular version of an apache mod for
> example, and a specific version off npm, which all only appeared in the
> ports tree at different times, so either we completely ditch the ports
> tree all together and import buildroot2 , which allows every package to
> have its own revision (but is Linux centric), or we need to generate
> frankenports trees. My curent iteration has 359 different packages, so
> you an imagine the hilarity  if I need to slide 20 of those back or
> forth, all independently.
> 
> There doesn't seem to be any understanding of this fact from the ports
> framework, and it means one has to keep one's own ports tree in source
> control, as a branch off the FreeBSD one. (maybe I should look at pkgsrc)..

I found this all confusing and vague, but it sounds like what's
happening is you need older versions of some software for whatever
reason and to provide that you are pulling older versions of ports into
a newer ports tree. Is that right?

> the problem is that the internal APIs of ports are changing too much.

Sorry, what do you mean? Can you define "too much" change?

> If you are going to change the API, then you need to be able to declare
> the version separately, maybe in a version/distinfo file that can be
> pulled in separately at a different rev, rather than having it built
> into the main Makefile of each port. Maybe the Makefile specifies a
> revision range it can be used with, but that would make a huge
> improvement right there.

The idea of having ports/packages support a version range for
dependencies is an interesting one, but I'm not sure how that would work.

> You can not just slide one port up by 3 months, and another down by 3
> months to get the revs one needs because the damned Mk files have
> changed. If I coudl leave the bulk of the Makefile alone and just slide
> the 'distingo/rev' file, (or be able to select a rev from one htat gives
> many alternatives, that woudl be "wonderful".

You're better off finding the tree that has the right version of the
oldest thing you need to use and then updating the things that you need
updated.

> Please, ports framework people,  think about how this can be done, If I
> have to edit a file, the game is lost.
> 
> Can we please get some understanding from ports people that they are
> making things REALLY HARD on vendors and it really strengthens the hands
> of  the "ditch FreeBSD and go to Linux" crowd when I need to spend a
> whole week trying to integrate in a set of 5 new ports into the product.
> 
> The call "It just works under linux, select the versions you want of
> each package and type make" is often heard around the company. And
> management is not totally deaf.
> 
> If you want to see how its' done better (still not perfect), go build a
> busybox system. you can effectively select any version of any tool and
> they all communicate via the pkgconf mechanism and you get the result
> you want.(mostly). And the API is stable.

Sounds like you've got a lot of folks who are used to the "LTS" mindset
where they never have to update their software to work with newer
versions. But ports is a rolling release in the head branch and
quarterly for those branches, and that's how it is going to be unless
someone wants to volunteer to maintain branches for longer. The LTS
thing works because people are paid to back port fixes and you can get
those for free.

> On the pkg side of things we need the ability for pkg to say "yeah I
> know I'm looking for foo-1.2.3.txz as a perfect match, but I've been
> given some slack on the third digit and I can see a foo-1.2.1.txz, so
> I'll install that instead".

I'm not sure how you would do that in a general way that didn't add
extra work for package maintainers.

> Otherwise we just have to spend WAY too much time generating dozens of
> "matching sets" of packages , that must be kept around forever if just
> one machine shipped with that set, not to mention the fact that making
> the matching set is often a real task.

Packages should generally be maintained as sets, rather than individual
packages, IMHO.

> The way to get around the problem above CAN be (not always) to install
> foo.1.2.1 first and THEN install the package you actually want, and pkg
> will accept it. The problem comes when pkg needs to install a dependency
> itself. Then it becomes "super picky", when there is actually a range of
> package revisions that would do. Instead of letting pkg install what it
> needs, we need to manually set up scripts to install the dependencies.
> so that all the work done by pkg is wasted.
> 
> 
> Please think about these two issues..

You can always use 'pkg lock' to lock a particular version of a package.

Steve

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 642 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-ports/attachments/20170209/a093d9dc/attachment.sig>