HEADS-UP: pkgbase pkg upgrade breaks systems Re: after updating to latest, pkg base segfaults and leaves me unbootable

Adding current@ to CC

On 09.08.25 01:38, Dan Mahoney wrote:
> 
> 
> (Resending from phone after realizing my list-specific from: wasn’t set, apologies for weird formatting)
> 
> Hey all,
> 
> After the recent big sleep in pkgbase, I hit the following trying to upgrade to whatever snapshot was published today:
> 
> [598/1127] Deleting files for p5-MIME-Base32-1.303: 100%
> [599/1127] Deinstalling p5-MIME-Base64-3.16...
> [599/1127] Deleting files for p5-MIME-Base64-3.16: 100%
> Child process pid=21537 terminated abnormally: Segmentation fault
> 
> (oh crap)
> 
> root@poudriere:/home/dmahoney # pkg upgrade
> ELF interpreter /libexec/ld-elf.so.1 not found, error 2
> Abort
> 
> (double crap)

Yes, I had the same issue yesterday evening. When the system was back, I 
was too tired to summarize and send something to the mailing list.

FWIW, I revived the system by deleting all newer packages rm 
/var/cache/pkg/*snap20250808* and just untaring the stuff in 
/var/cache/pkg to /

cd / ; for i in `/rescue/ls -1 /var/cache/pkg/FreeBSD-* ` ; do 
/rescue/tar xvzf $i ; done

After that, I resorted to building from source and installing to get the 
system into a half way consistent state with a chance of surviving a reboot.

When starting my upgrade I saw that it wanted to remove a lot of non 
pkgbase packages (how are we doing to differentiate pkgbase packages and 
"ports" packages in the future?). I thought this might be related to the 
krb5 thing, so I created an up to date poudriere jail via pkgbase method 
and rebuilt all my pkgs, but even then I saw the same thing as Den, that 
pkg wanted to remove a lot of ports pkgs, as this system is not 
important I thought I can resolve that after the pkgbase upgrade and 
started the upgrade.

I didn't save scroll back. In my case I saw at the top the first ~100 
pkg transactions were uninstalling pkgbase pkgs, then it upgraded some, 
then pkg exited with a segfault.

Leaving me with ELF interpreter /libexec/ld-elf.so.1 not found, error 2

One thing I checked was /libexec/ was completely empty.

Florian

> 
> root@poudriere:/home/dmahoney # pkg-static install -f pkg
> pkg-static: Unable to determine the ABI, none of the ABI_FILEs can be read.
> pkg-static: Cannot parse configuration file!
> root@poudriere:/home/dmahoney # /rescue/sh
> Cannot read termcap database;
> using dumb terminal settings.
> # pkg-static
> pkg: not enough arguments
> Usage: pkg [-v] [-d] [-l] [-N] [-j <jail name or id>|-c <chroot path>|-r <rootdir>] [-C <configuration file>] [-R <repo config dir>] [-o var=value] [-4|-6] <command> [<args>]
> 
> For more information on available commands and options see 'pkg help'.
> # pkg-static install -f pkg
> pkg-static: Unable to determine the ABI, none of the ABI_FILEs can be read.
> 
> Now, if this were 14.x, I'd fix this by untarring a distfile right over / to get back up and running.  Not quite an option in 15, is it?  (I would encourage the people making this stuff to please consider keeping those built, even if bsdinstall uses pkgbase).
> 
> This is dayjob's poudriere system -- the build process for it is well documented and the parts of it that aren't managed by puppet are easily managed because I captured every command used to create every jail and ports tree (which was slow because many of them came from freebsd-archive; both old jails and old copies of ports trees last known to work with given versions of FreeBSD (kind of required when you needed to build packages in 2020 because remote hands were Not An Option).  Like all our systems, the bits we care about (homedirs, /usr/local/etc) are in backups as well.  We kind of need it to work.
> 
> So this isn't an emergency.  This system is really there so that me, as a port maintainer, can build a debug build of something, but this machine isn't in our critical path.  ...but What If It Was?  We run Critical Stuff, out there on lone servers in faraway places (on bare metal)
> 
> But it's *really* not instilling me with a lot of confidence in the readiness of this pkgbase idea.  (For the record, I've also had freebsd-update leave me dead on the table in similar ways in the past.  I literally called them out during a BSDcan talk without trying to bash Colin too hard).
> 
> I can capture more scrollback if people want, but I wasn't doing any of the crazy -f commands people are talking about.  This was literally a "pkg upgrade".
> 
> Full command output is over at https://users.isc.org/~dmahoney/failedupgrade.txt if devs want to have a look and try to black-box it.  I'll keep the VM running (and logged in, in a screen session) if there are things people want me to try.
> 
> This begs the questions:
> 
> * Is there CI for pkgbase that tries to upgrade from whatever version is immediately previous to it, before publishing it?  (I know that's what version I was running, it's the version everyone's been stuck at for weeks!)
> 
> * For all the debate about "pkgbase and pkg should be exactly the same", perhaps pkgbase could have an auto-bectl in it?
> 
> * Is there support somewhere for having a lockstep "set" of packages that one knows are in /var/db/pkg and constitute a relatively concurrent install?  (Or a list of files that I could tell pkg-static to install with a glob, out of /var/cache/pkg)
> 
> * With something like -CURRENT, is there any support for saying "Okay not the current-current tree, but current-minus-one"  (I guess that would be if you're working with the weekly builds, but that's not quite the same.  Your only option is "base system of the now" but it happens less frequently).
> 
> * And for running -CURRENT where this kind of breakage can happen, could we get a statically linked version of pkg?
> 
> Any questions, let me know.
> 
> -Dan
> 
> Sent from my iPhone
>