Ports tree gone unstable?

Don Lewis truckman at FreeBSD.org
Thu Apr 28 00:17:18 UTC 2016

On 28 Apr, Michelle Sullivan wrote:
> Don Lewis wrote:
>> On 27 Apr, Michelle Sullivan wrote:
>>> Don Lewis wrote:
>>>> On 27 Apr, Michelle Sullivan wrote:
>>>>> Don Lewis wrote:
>>>>>> On 27 Apr, Michelle Sullivan wrote:
>>>>>>> Don Lewis wrote:
>>>>>>>> On 27 Apr, Rick Miller wrote:
>>>>>>>>> On Wed, Apr 27, 2016 at 12:53 PM, Michelle Sullivan <michelle at sorbs.net>
>>>>>>>>> wrote:
>>>>>>>>>> Kevin Oberman wrote:
>>>>>>>>>>> On Wed, Apr 27, 2016 at 8:06 AM, Michelle Sullivan <michelle at sorbs.net>
>>>>>>>>>>> wrote:
>>>>>>>>>>> After a portsnap update it seems all my jails won't build the current tree
>>>>>>>>>>>> returning the following error:
>>>>>>>>>>>> ====>> MOVED: sysutils/puppet renamed to sysutils/puppet38
>>>>>>>>>>>> ====>> MOVED: textproc/rubygem-augeas renamed to
>>>>>>>>>>>> textproc/rubygem-ruby-augeas
>>>>>>>>>>>> ====>> Computing deps for converters/libiconv
>>>>>>>>>>>> ====>> Computing deps for archivers/unzip
>>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found.
>>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found.
>>>>>>>>>>>> ====>> Computing deps for converters/p5-Encode
>>>>>>>>>>>> ====>> Computing deps for converters/p5-Convert-BinHex
>>>>>>>>>>>> ====>> Computing deps for converters/p5-Encode-Locale
>>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found.
>>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found.
>>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found.
>>>>>>>>>>>> ====>> Computing deps for converters/p5-JSON-PP
>>>>>>>>>>>> ====>> Computing deps for converters/p5-JSON
>>>>>>>>>>>> ====>> Computing deps for converters/p5-JSON-XS
>>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found.
>>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found.
>>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found.
>>>>>>>>>>>> ====>> Computing deps for converters/p5-Text-Iconv
>>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found.
>>>>>>>>>>>> ====>> Computing deps for databases/ip4r
>>>>>>>>>>>> ====>> Computing deps for databases/gdbm
>>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found.
>>>>>>>>>>>> ====>> Computing deps for databases/p5-Bucardo
>>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found.
>>>>>>>>>>>> Terminated
>>>>>>>>>>>> Terminated
>>>>>>>>>>>> Terminated
>>>>>>>>>>>> Terminated
>>>>>>>>>>>> ====>> Cleaning up
>>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found.
>>>>>>>>>>>> ====>> Computing deps for databases/p5-DBD-Pg
>>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found.
>>>>>>>>>>>> ====>> Computing deps for databases/memcached
>>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/automake-1.15' not
>>>>>>>>>>>> found.
>>>>>>>>>>>> ====>> Umounting file systems
>>>>>>>>>>>> Checked updating but don't see anything to suggest that port origins of
>>>>>>>>>>>> '/usr/local/bin/ccache' are normal..
>>>>>>>>> It looks like you're building with Poudriere.  I observed similar behavior,
>>>>>>>>> but not the exact message the other day.  I don't remember what origin it
>>>>>>>>> was complaining about, but located a post (either on a mailing list or
>>>>>>>>> forums) recommending a `pkg install poudriere`.  It did resolve the problem
>>>>>>>>> in this particular scenario.
>>>>>>>> This is probably caused by the recent change to globally drop
>>>>>>>> ${PORTSDIR} from *_DEPENDS.  The framework changes initially were done
>>>>>>>> in bsd.port.mk r399278, but the the actual removal of ${PORTSDIR} didn't
>>>>>>>> happen until r411970, r412342, ...
>>>>>>> Ok that sorta makes a bit more sense... however as this is a jail and
>>>>>>> the tree is updated why did it break?  (I have no local mods in the 'ng'
>>>>>>> build tree - except an additional (local only) couple of ports which are
>>>>>>> copied in manually after the portsnap update)...
>>>>>>> Of course the nice thing is my non-ng tree is still working 100% - but
>>>>>>> that would be because it didn't get the change... but again that's a
>>>>>>> completely separate tree and the 2 are not associated with each other in
>>>>>>> any way...
>>>>>> I was assuming that this was your non-ng tree where you have local
>>>>>> framework changes ...
>>>>> No, completely separate repo as the new trees are constantly breaking my
>>>>> tree so I keep them entirely separate.
>>>>>> Did you upgrade ports from something older than r411970 (Sun Mar 27
>>>>>> 01:23:25 2016 UTC)
>>>>> It would have been around march I did the last build so yes probably
>>>>> prior to Mar 27.
>>>>>>     to something more recent?
>>>>> To the latest.
>>>>>>     If poudriere on your host
>>>>>> is seriously old, it might not cope with the framework change.  It looks
>>>>>> like you need at least 3.1.9, which was released on Wed Oct 14 21:06:00
>>>>>> 2015 UTC.
>>>>> Yeah, 3.1.x changes the base OS without authority and breaks the entire
>>>>> build system (can't build anything but the official tree in it) so it's
>>>>> been deemed a security issue (because it "upgrades" the existing
>>>>> repositories) and therefore cannot be installed or used on any of the
>>>>> existing build servers.
>>>> Sorry, this is all from memory ... my poudriere machine will be offline
>>>> for several more hours so I can't use it as a reference.
>>>> Poudriere shouldn't be changing anything in the base OS.  It probably
>>>> creates some temp files under /tmp and puts the package repositories
>>>> that I builds and the log files under its own directories under
>>>> /var/tmp.
>>> Unfortunately the first thing that 3.1.[012]? did was install all the
>>> pkg stuff and change the pkg_add repo into a pkg repo... or something
>>> like it, which broke everything horribly..  it was a long time ago so no
>>> idea the specifics now... but it (3.1.x) was put on a 'not suitable for
>>> use due to security issues' list.
>> That could be the point at which support for the old pkg_* tools was
>> removed from poudriere.  That would explain why it wanted to upgrade
>> things.
> Yes, but that's also a security issue in itself.  Had I not spotted and 
> stopped this the changes it tried to make to the repo would have 
> resulted in production machines being 'reload from scratch' on the first 
> security patch - because it screwed the build servers and the repos.

I would think that the pkg_* tools would see the NG repo as something
that they didn't understand and then would barf without doing anything.

>> I'd thought that poudriere was using the host copy of pkg to do the
>> final part of the respository build, but since poudriere doesn't list
>> pkg as a dependency, that appears not to be the case.  It looks like
>> poudriere is running pkg (from the repository being constructed) in the
>> jail for that.
> Yeah.. except something around the time went about "upgrading" the OS to 
> use pkg as well... which screwed the OS... fortunately I caught the 
> first VM it tried to do it to and was able to limit the damage just to 
> that VM so the rebuild was minimum.

That's quite possible.  If the OS is old enough to have been using the
old pkg_* tools by default, then it would have gotten an update to
switch to pkgNG when the support for old-style packages was removed from
the ports tree.  Prior to that, there was a lengthy period of time when
old packages were the default, but you could switch to new packages by
adding some magic to /etc/make.conf.  I thought all the magic was in the
ports framework and prior to the cutover it just looked at ${OSVERSION},
though.  A client without a copy of /usr/ports wouldn't know about any
of this.

I would think that the old pkg_* tools would still work with an
old-style repository, but that's not something that I've tried.

>>>> You should be able to build as many different ports trees as you want
>>>> and they can be downloaded via portsnap or svn, or created by hand.
>>>> I think I've got 4-6 ports trees that I use with poudriere.
>>> Which I have 2 currently - one which is 'HEAD' and the other which is my
>>> 'pkg_*' tools tree (up to date - mostly).
>>>> The repository that gets updated by a poudriere run is named
>>>> with a combination of the jail name, the ports tree name, and the set
>>>> name (-z option).  The latter can be use to select an alternate
>>>> make.conf to set different port options.
>>> Yes, however 3.1.x 'updated' the repo from pkg_* to something like pkgng
>>> - it was completely f**ked though... basically had to erase everything,
>>> downgrade and reinstall everything to get it back to a 'will build both
>>> trees' state.
>>>>> Question is why would it be needed?  Surely the tree is the tree in the
>>>>> jail and has nothing to do with the host?  or is it not a case of
>>>>> everything is done in the jail, just the actual building is and
>>>>> therefore I need new build servers for the NG tree.. Which basically
>>>>> means I should just decide to fork or erase the whole system because I
>>>>> can't "NG" right now and I can't actually continue to build in parallel
>>>>> because of this breakage?
>>>> Only the actual building is done in jails.  When poudriere first starts
>>>> up, it looks at the list of ports that you want to build and then uses
>>>> the Makefiles for each of those ports to determine the dependencies of
>>>> each and the proper build order.
>>> That doesn't make sense...  the host has for months been well behind
>>> both my tree and the ng trees... all the versions would be way out of
>>> whack and even some not existing (ruby comes to mind.)
>> As I mentioned previously, poudriere is just a shell script so it isn't
>> tightly bound to the version of the ports tree.  What caused the problem
>> that you are seeing now is that the ports framework changed in a way
>> that is not compatible with really old versions of poudriere.  Basically
>> the output of "make -V BUILD_DEPENDS" changed from
>> blah:/path/to/ports/category/port to blah:category/port, and the old
>> version of poudriere can't cope with that.
> I find that 'odd' at first parse... though will take your word for it 
> and check the code... maybe make my own version of poudriere that can 
> handle both...

With a ports tree from late last year:

%cd /usr/ports/lang/gcc
/usr/local/bin/as:/usr/ports/devel/binutils gmake:/usr/ports/devel/gmake libiconv>=1.14_9:/usr/ports/converters/libiconv /usr/local/share/java/ecj-4.5.jar:/usr/ports/lang/gcc-ecj45  zip:/usr/ports/archivers/zip /usr/local/bin/as:/usr/ports/devel/binutils perl5>=5.20<5.21:/usr/ports/lang/perl5.20

>From a recent ports tree:

/usr/local/bin/as:devel/binutils gmake:devel/gmake libiconv>=1.14_9:converters/libiconv makeinfo:print/texinfo /usr/local/share/java/ecj-4.5.jar:lang/gcc-ecj45  zip:archivers/zip /usr/local/bin/as:devel/binutils perl5>=5.20<5.21:lang/perl5.20

It turns out that there are more moving parts in poudriere that I
expected.  Patching your copy of poudriere looks like your best bet.
That will also allow you to build old-style packages with recent
versions of the ports tree.  Look for changes between version 3.1.8 and
3.1.9 here: <https://github.com/freebsd/poudriere/>

>>>> The ports tree used by each build jail is not related to any ports tree
>>>> on the host (unless you do something like "poudriere ports -c -F -f none
>>>> -M /usr/ports -p systemports", see
>>>> <https://fossil.etoilebsd.net/poudriere/doc/trunk/doc/use_system_ports_tree.wiki>).
>>>> It's possible to run poudriere without having /usr/ports installed on
>>>> the host.
>>> Will check this - if it mounts the local copy of the tree this would
>>> probably fix it.
>>>> If you think this is a security risk, you could run poudriere in a VM.
>>> They already are in VMs.. but if poudriere make modifications to the OS
>>> then it is a security issue.  If it modifies/builds packages that's fine.
>>> ... let me expand on that ... anything that modifies something in the
>>> base OS unless specifically designed and approved to interact with the
>>> OS (eg puppet) then as far as I am concerned (and my employer) it's a
>>> security risk.  Can't have things willy nilly changing the OS, it will
>>> eventually break stuff and that could cause/lead to production
>>> outages... it's just not done.
>> I don't think it's changing anything in the base OS.
> Well bapt swears that it wasn't freebsd-update that screwed the OS, so 
> must be pourdiere ... don't know both changes happened at the same time, 
> and it screwed my migration plans to the point I've told $employer to 
> forget looking at FreeBSD at all for internal systems or embedded 
> systems and stick with the current OS of RHEL/CentOS and please assign 
> people to take over my machines and change them to match the corp 
> standard...  So far they have refused to switch them (lack of resources) 
> and after a long discussion with a person in OPs that also has an 
> @freebsd.org address it was suggested I just fork the OS and ports 
> tree...  However, after news that i might not have as long as I hoped, I 
> think this would be folly and a waste of my remaining days.

Funny thing that you should mention freebsd-update ... in another thread
a couple people mentioned getting the wrong version of ifconfig after
running freebsd-update.

>>   It sounds to me
>> like the problem is confined to upgrading old-style pkg repositories to
>> the NG when you try to update some of the packages contained in them,
>> which is not what you want in this case.
> Yeah that happened as well.
>> You should be able to install a newer version of poudriere alongside the
>> old one.
> That occurred to me, however unless its changed a lot it'll really not 
> work unless chrooted elsewhere because of shared scripts... or 
> rebuilding it to handle the shared stuff in different places (assuming 
> not already able to cope)
>>   Just call it poudriere-ng or something.  Just be sure not to
>> run it on one of your old pkg repositories.
> That's not going to be an issue...  will probably just have to put in 
> new VMs... just about got everything off 9.1 and pretty sure nothing is 
> on 9.0 now so maybe re-purpose them to be NG only hosts.
>>    You might even be able to
>> edit the script to bail out if it detects and old-style pkg repository
>> as an anti-foot-shooting measure.
> Think it would need to be physically separated because I know if bapt 
> was not lying about freebsd-update making the changes then it must have 
> been poudriere that changed the OS... so not worth taking the risk.
> For the time being, just disable to entire NG repo keep my fork going 
> with the working (mostly) up to date ports tree for pkg_* tools and well 
> if the worst comes to the worst it'll fall to someone else in my absence 
> and they can deal with switching to RHEL/CentOS because they'd never 
> work out how the SORBS FreeBSD build system is put together.
> Thanks for the replies though Don, all the best.

More information about the freebsd-ports mailing list