Ports tree gone unstable?

[Don Lewis]

On 28 Apr, Michelle Sullivan wrote:
> Don Lewis wrote:

>> I'd thought that poudriere was using the host copy of pkg to do the
>> final part of the respository build, but since poudriere doesn't list
>> pkg as a dependency, that appears not to be the case.  It looks like
>> poudriere is running pkg (from the repository being constructed) in the
>> jail for that.
> 
> Yeah.. except something around the time went about "upgrading" the OS to 
> use pkg as well... which screwed the OS... fortunately I caught the 
> first VM it tried to do it to and was able to limit the damage just to 
> that VM so the rebuild was minimum.

How were you upgrading ports when this happened.  I ask because the old
pkg_* tools didn't have the equivalent of "pkg upgrade".  I'm guessing
that you probably used "portupgrade -P" or "portmaster -P" as a wrapper
around the pkg_* tools, and I think that requires a copy of the ports
tree on the client machines even though binary packages are being used.

If that's the situation, I think what probably happened is that when up
updated to a version of the ports tree after support for old-style
packages was turned off, poudriere rebuilt the repository with new-style
packages.  Then when you ran portupgrade or portmaster on the client
with the same version of the ports tree, the framework then told
portmaster/portupgrade to not look for WITH_PKGNG=yes in
/etc/make.conf and just to go ahead and use pkg to upgrade the packages.
Since the database of installed packages hadn't been upgraded with
pkg2ng, chaos ensued.


At the time of this transition, I was still using portupgrade to build
everything from source.  Before I switched to pkgNG, I was having
problems with database corruption because the old package tools didn't
really handle running out of disk space during an upgrade.  I only
upgraded packages infrequently because the process was so painful.  It
would take two or three days to do an upgrade on my desktop and I'd have
to monitor it around the clock.  If portupgrade ran into an error or
stopped to ask a question just after I went to sleep, then I'd lose many
hours of potential build time.  Because of the infrequent upgrades I
would have to deal with all of the intervening special cases in UPDATING
that accumulated between upgrades, and the portupgrade -fr and -a
options didn't interoperate well, so I ended up having to build some
ports multiple times.  If things crashed, then I'd have to run
portupgrade -rf again, rebuilding a lot of things unnecessarily since
there was no way of doing a restart.  A classic cause of that would
happen if portupgrade decided to rebuild gdm, in which case it would
stop gdm before removing the the old version and restart it after
installing the new version.  Unfortunately, stopping gdm would kill Xorg
and thus the terminal window where portupgrade was running.

Eventually things got to the point that I could no longer tolerate the
extended downtime of my primary desktop machine, so I started building
binary packages using portupgrade -p on a faster headless machine.  The
builds still took a long time, but the final upgrade on my desktop using
"pkg upgrade" was *much* faster.  Building the packages with portupgrade
was still flakey and eventually broke when the ports tree was converted
to staging.   At that point I bit the bullet and converted to poudriere
and life was so much better.  Not only did that eliminate a lot of
manual intervention to build the packages, but the paralled builds sped
things up a lot.  Even though poudriere isn't especially efficient about
deciding what needs to be rebuilt, the build times for my package set
went from several days to under 12 hours, and the latter includes a
number of huge ports that I never used to build.