Ports tree gone unstable?

Thu Apr 28 10:49:15 UTC 2016

Don Lewis wrote:
> On 28 Apr, Michelle Sullivan wrote:
>> Don Lewis wrote:
>>> I'd thought that poudriere was using the host copy of pkg to do the
>>> final part of the respository build, but since poudriere doesn't list
>>> pkg as a dependency, that appears not to be the case.  It looks like
>>> poudriere is running pkg (from the repository being constructed) in the
>>> jail for that.
>> Yeah.. except something around the time went about "upgrading" the OS to
>> use pkg as well... which screwed the OS... fortunately I caught the
>> first VM it tried to do it to and was able to limit the damage just to
>> that VM so the rebuild was minimum.
> How were you upgrading ports when this happened.  I ask because the old
> pkg_* tools didn't have the equivalent of "pkg upgrade".  I'm guessing
> that you probably used "portupgrade -P" or "portmaster -P" as a wrapper
> around the pkg_* tools, and I think that requires a copy of the ports
> tree on the client machines even though binary packages are being used.

Actually no, I was 'pkg_delete -fa' followed by a list of pkg_add -r 
http:// ... on the build hosts...  Puppet deals with the production 
servers and relies on the entire repo list being successfully built and 
then all the packages being successfully regression tested.

>
> If that's the situation, I think what probably happened is that when up
> updated to a version of the ports tree after support for old-style
> packages was turned off, poudriere rebuilt the repository with new-style
> packages.  Then when you ran portupgrade or portmaster on the client
> with the same version of the ports tree, the framework then told
> portmaster/portupgrade to not look for WITH_PKGNG=yes in
> /etc/make.conf and just to go ahead and use pkg to upgrade the packages.
> Since the database of installed packages hadn't been upgraded with
> pkg2ng, chaos ensued.
>
>
> At the time of this transition, I was still using portupgrade to build
> everything from source.  Before I switched to pkgNG, I was having
> problems with database corruption because the old package tools didn't
> really handle running out of disk space during an upgrade.  I only
> upgraded packages infrequently because the process was so painful.  It
> would take two or three days to do an upgrade on my desktop and I'd have
> to monitor it around the clock.  If portupgrade ran into an error or
> stopped to ask a question just after I went to sleep, then I'd lose many
> hours of potential build time.  Because of the infrequent upgrades I
> would have to deal with all of the intervening special cases in UPDATING
> that accumulated between upgrades, and the portupgrade -fr and -a
> options didn't interoperate well, so I ended up having to build some
> ports multiple times.  If things crashed, then I'd have to run
> portupgrade -rf again, rebuilding a lot of things unnecessarily since
> there was no way of doing a restart.  A classic cause of that would
> happen if portupgrade decided to rebuild gdm, in which case it would
> stop gdm before removing the the old version and restart it after
> installing the new version.  Unfortunately, stopping gdm would kill Xorg
> and thus the terminal window where portupgrade was running.
>
> Eventually things got to the point that I could no longer tolerate the
> extended downtime of my primary desktop machine, so I started building
> binary packages using portupgrade -p on a faster headless machine.  The
> builds still took a long time, but the final upgrade on my desktop using
> "pkg upgrade" was *much* faster.

I went with pkg on one of my staging servers for the prod environment... 
the server had to be wiped and reinstalled from scratch... I haven't go 
with pkgng since and have nicknamed it 'pkg no good'.. figured I'd give 
it time to actually become stable before trying again (you know where 
you don't get an announcement that you have to upgrade past "this" 
version to make stuff work again).. however it's rapidly becoming 
apparent that the systems will no longer build without completely 
switching everything so its go with the unstable versions or drop NG 
support until I have time to fix what was broken...  Health comes first, 
NG is officially dropped across the company as a whole it won't be 
happening now until/if I'm given the all clear.

>    Building the packages with portupgrade
> was still flakey and eventually broke when the ports tree was converted
> to staging.   At that point I bit the bullet and converted to poudriere
> and life was so much better.  Not only did that eliminate a lot of
> manual intervention to build the packages, but the paralled builds sped
> things up a lot.  Even though poudriere isn't especially efficient about
> deciding what needs to be rebuilt, the build times for my package set
> went from several days to under 12 hours, and the latter includes a
> number of huge ports that I never used to build.
>

I used to patch by hand as needed.  I switch to a load of headless VMs, 
jails, Jenkins, Puppet etc when there was the 'staging' push when I 
thought I could do my bit for the project.. little was I to know how 
little certain people care about people giving their all or production 
servers outside of their little domain of desktop machines who were 
pushing policy so fast regardless of the real world and regardless of 
current critical security issues that it made it impossible to switch 
before the deadline (asside from the fact unless you are part of the 
"leet" group in the know, the switch date was indicated just as an EOL 
(things would not be supported after this date), rather than a deadline 
of switch or your system will be broken the next time you try to do 
anything... and that the some critical patches were pushed to after the 
EOL and were never rolled into the quarterly.)

So I set about creating a ports tree that works with the old tools and 
builds with the pkgng stuff... guess what... it still works.. the only 
thing that broke is the raw FreeBSD ports tree from HEAD...  Funny thing 
is I'd almost got the tool chain building the latest ports on 6.x and 
7.x.. even pkgng on 6.x until I found that the only thing stopping pkgng 
on 6.x was the use of dprinf() .. which I could duplicate a macro for, 
but I though, why they f**k should I bother... I have my tree working 
for pkg_* which means I can patch the legacy systems.  ...and the 
unofficial dropping of support for i386 makes FreeBSD more and more 
irrelevant as far as I am concerned...

Regards,

Michelle

("unofficial dropping" - the responses for bugs have been "duplicate on 
64bit as it's old (legacy) hardware" - eg (probably a bad example, but 
sufficient for this example) zfs bugs get the general response 'don't do 
zfs on i386 its not supported' and yet it's still presented as an option 
for the root fs at install, and it's still compiled in by default on 
i386... there are better examples, but I can't be bothered to look them 
up as it's a waste of time, you might be having an intelligent 
conversation with me but others won't.)

-- 
Michelle Sullivan
http://www.mhix.org/