Re: HEADS-UP: pkgbase pkg upgrade breaks systems Re: after updating to latest, pkg base segfaults and leaves me unbootable

> On Aug 9, 2025, at 12:55 AM, Florian Smeets <flo@smeets.xyz> wrote:
> 
> 
> FWIW, I revived the system by deleting all newer packages rm /var/cache/pkg/*snap20250808* and just untaring the stuff in /var/cache/pkg to /
> 
> cd / ; for i in `/rescue/ls -1 /var/cache/pkg/FreeBSD-* ` ; do /rescue/tar xvzf $i ; done
> 
> After that, I resorted to building from source and installing to get the system into a half way consistent state with a chance of surviving a reboot.

So, looking at Florian's solution, I came across two issues.

1) After you build from src, pkg thinks you have one thing installed, and buildworld has installed something different.  I cannot figure out how to tell pkg "delete anything you know from the pkgdb about these pkgs, and just let me re-run pkgbasify".  There's a -O option that exists in netbsd's pkg, and I'm sure there's some absurd invocation that involves manipulating the raw pkgdb, but nothing "easy"/

2) Pkgbase got me into this problem, and I wanted it to get me out.  I had all the pkgs sitting there in /var/cache/pkg, ready to go, I just needed to get them in there.   Doing a pkg-info yielded a scant 20 packages or so.

Worse still: /rescue/ls /var/cache/pkg/FreeBSD-* gave me "argument list too long" and with no grep, I basicaly had to rely on scrollback to get what I was looking for.

Ergo, my solution was more like:

cd /var/cache
/rescue/cp -pR pkg pkg-backup
mkdir ../pkg-snap20250808125411
cp FreeBSD*snap20250808125411* ../pkg-snap20250808125411

(From there, I used rm cautiously to remove all the -dbg packages -- I can put them back later, but for right now I needed them to work.

Let's just try one package...

# ABI=FreeBSD:15:amd64 /usr/local/sbin/pkg-static add -M ./FreeBSD-libarchive-15.snap20250808125411.pkg
pkg-static: Setting ABI requires setting OSVERSION, guessing the OSVERSION as: 1500000
Installing FreeBSD-libarchive-15.snap20250808125411...
Newer FreeBSD version for package FreeBSD-libarchive:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1500058
- running userland: 1500000
Ignore the mismatch and continue? [y/N]: y
`-- Installing FreeBSD-runtime-15.snap20250808125411...
|   `-- Installing FreeBSD-libarchive-15.snap20250808125411...
|   | `-- Installing FreeBSD-runtime-15.snap20250808125411...
|   |   `-- Installing FreeBSD-libarchive-15.snap20250808125411...
|   |   | `-- Installing FreeBSD-runtime-15.snap20250808125411...
|   |   |   `-- Installing FreeBSD-libarchive-15.snap20250808125411...
|   |   |   | `-- Installing FreeBSD-runtime-15.snap20250808125411...
|   |   |   |   `-- Installing FreeBSD-libarchive-15.snap20250808125411...
^C

(Okay, so there's no "y" option to pkg add... and these things will forever sit and try to install circular dependencies.  Dumb).

New solution: What if we're in a new directory that only has packages in it that have already been installed?

mkdir /tmp/foo
cd /var/cache/pkg-snap20250808125411/
cp FreeBSD-libarchive-15.snap20250808125411.pkg /tmp/foo
cd /tmp/foo
ABI=FreeBSD:15:amd64 /usr/local/sbin/pkg-static add -M ./FreeBSD-libarchive-15.snap20250808125411.pkg
pkg-static: Setting ABI requires setting OSVERSION, guessing the OSVERSION as: 1500000
Installing FreeBSD-libarchive-15.snap20250808125411...
Newer FreeBSD version for package FreeBSD-libarchive:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1500058
- running userland: 1500000
Ignore the mismatch and continue? [y/N]: y
pkg-static: Missing shlib dependency: libbsdxml.so.4
pkg-static: Missing shlib dependency: libbz2.so.4
pkg-static: Missing shlib dependency: libcrypto.so.30
pkg-static: Missing shlib dependency: liblzma.so.5
pkg-static: Missing shlib dependency: libprivatezstd.so.5
pkg-static: Missing shlib dependency: libz.so.6
Extracting FreeBSD-libarchive-15.snap20250808125411: 100%

Okay, but this will still be exhausting if I need to keep hitting "yes"...

for i in `ls -1 /var/cache/pkg-snap20250808125411/`
> do
> cp /var/cache/pkg-snap20250808125411/$i /tmp/foo
> cd /tmp/foo
> OSVERSION=1500058 ABI=FreeBSD:15:amd64 /usr/local/sbin/pkg-static add -M ./$i
> done


Installing FreeBSD-acct-15.snap20250808125411...
pkg-static: Missing shlib dependency: libxo.so.0
Extracting FreeBSD-acct-15.snap20250808125411: 100%
Installing FreeBSD-acct-man-15.snap20250808125411...
Extracting FreeBSD-acct-man-15.snap20250808125411: 100%
Installing FreeBSD-acpi-15.snap20250808125411...
[...]
Installing FreeBSD-zfs-15.snap20250808125411...
Extracting FreeBSD-zfs-15.snap20250808125411: 100%
Installing FreeBSD-zfs-dev-15.snap20250808125411...
Extracting FreeBSD-zfs-dev-15.snap20250808125411: 100%
Installing FreeBSD-zfs-dev-lib32-15.snap20250808125411...
pkg-static: Missing shlib dependency: libicp.so.3:32
pkg-static: Missing shlib dependency: libzfsbootenv.so.1:32
Extracting FreeBSD-zfs-dev-lib32-15.snap20250808125411: 100%
Installing FreeBSD-zfs-lib32-15.snap20250808125411...
Extracting FreeBSD-zfs-lib32-15.snap20250808125411: 100%
Installing FreeBSD-zfs-man-15.snap20250808125411...
Extracting FreeBSD-zfs-man-15.snap20250808125411: 100%
Installing FreeBSD-zoneinfo-15.snap20250808125411...

...I started a normal shell (/bin/csh) and tried a service command, and sshd restarted.

But pkg is still being persnickety, and I don't know what the mechanism is to bootstrap it is:

# ldd /usr/local/sbin/pkg
/usr/local/sbin/pkg:
        libelf.so.2 => /lib/libelf.so.2 (0x2a08ac8d1000)
        libjail.so.1 => /lib/libjail.so.1 (0x2a08ad283000)
        libm.so.5 => /lib/libm.so.5 (0x2a08adcbd000)
        libarchive.so.7 => /usr/lib/libarchive.so.7 (0x2a08ae37a000)
        libz.so.6 => /lib/libz.so.6 (0x2a08abe01000)
        libutil.so.9 => not found (0)
        libssl.so.30 => /usr/lib/libssl.so.30 (0x2a08af3bb000)
        libcrypto.so.30 => /lib/libcrypto.so.30 (0x2a08af8a2000)
        libthr.so.3 => /lib/libthr.so.3 (0x2a08b0575000)
        libc.so.7 => /lib/libc.so.7 (0x2a08b19a8000)
        libbz2.so.4 => /usr/lib/libbz2.so.4 (0x2a08b257a000)
        liblzma.so.5 => /usr/lib/liblzma.so.5 (0x2a08b2d8e000)
        libbsdxml.so.4 => /lib/libbsdxml.so.4 (0x2a08b3302000)
        libprivatezstd.so.5 => /usr/lib/libprivatezstd.so.5 (0x2a08b3d33000)
        libsys.so.7 => /lib/libsys.so.7 (0x2a08b4a90000)
        libmd.so.7 => /lib/libmd.so.7 (0x2a08b4f76000)
        [vdso] (0x2a08ab287000)

I've got a libutil.so.10, but not 9.  Doing a pkg-static install -f pkg doesn't update it to use .so.10 either.  Where does one get this file?

====

I don't have another system with pkgbase on it, so I can't ask what pkg provides that file.  I don't know if, given a list of .pkg files, there's a way to poll them all and ask.  (pkg grep?)

While this is still a prerelease thing -- these recipes should be in the pkgbase wiki page.  Knowing how to recover from a pkgbase failure -- or how to rectify a world and pkgbase that are out of step with each other, are going to be things we're going to need to do occasionally.

> 
> When starting my upgrade I saw that it wanted to remove a lot of non pkgbase packages (how are we doing to differentiate pkgbase packages and "ports" packages in the future?). I thought this might be related to the krb5 thing, so I created an up to date poudriere jail via pkgbase method and rebuilt all my pkgs, but even then I saw the same thing as Den, that pkg wanted to remove a lot of ports pkgs, as this system is not important I thought I can resolve that after the pkgbase upgrade and started the upgrade.
> 
> I didn't save scroll back. In my case I saw at the top the first ~100 pkg transactions were uninstalling pkgbase pkgs, then it upgraded some, then pkg exited with a segfault.
> 
> Leaving me with ELF interpreter /libexec/ld-elf.so.1 not found, error 2
> 
> One thing I checked was /libexec/ was completely empty.
> 
> Florian
> 
>> root@poudriere:/home/dmahoney # pkg-static install -f pkg
>> pkg-static: Unable to determine the ABI, none of the ABI_FILEs can be read.
>> pkg-static: Cannot parse configuration file!
>> root@poudriere:/home/dmahoney # /rescue/sh
>> Cannot read termcap database;
>> using dumb terminal settings.
>> # pkg-static
>> pkg: not enough arguments
>> Usage: pkg [-v] [-d] [-l] [-N] [-j <jail name or id>|-c <chroot path>|-r <rootdir>] [-C <configuration file>] [-R <repo config dir>] [-o var=value] [-4|-6] <command> [<args>]
>> For more information on available commands and options see 'pkg help'.
>> # pkg-static install -f pkg
>> pkg-static: Unable to determine the ABI, none of the ABI_FILEs can be read.
>> Now, if this were 14.x, I'd fix this by untarring a distfile right over / to get back up and running.  Not quite an option in 15, is it?  (I would encourage the people making this stuff to please consider keeping those built, even if bsdinstall uses pkgbase).
>> This is dayjob's poudriere system -- the build process for it is well documented and the parts of it that aren't managed by puppet are easily managed because I captured every command used to create every jail and ports tree (which was slow because many of them came from freebsd-archive; both old jails and old copies of ports trees last known to work with given versions of FreeBSD (kind of required when you needed to build packages in 2020 because remote hands were Not An Option).  Like all our systems, the bits we care about (homedirs, /usr/local/etc) are in backups as well.  We kind of need it to work.
>> So this isn't an emergency.  This system is really there so that me, as a port maintainer, can build a debug build of something, but this machine isn't in our critical path.  ...but What If It Was?  We run Critical Stuff, out there on lone servers in faraway places (on bare metal)
>> But it's *really* not instilling me with a lot of confidence in the readiness of this pkgbase idea.  (For the record, I've also had freebsd-update leave me dead on the table in similar ways in the past.  I literally called them out during a BSDcan talk without trying to bash Colin too hard).
>> I can capture more scrollback if people want, but I wasn't doing any of the crazy -f commands people are talking about.  This was literally a "pkg upgrade".
>> Full command output is over at https://users.isc.org/~dmahoney/failedupgrade.txt if devs want to have a look and try to black-box it.  I'll keep the VM running (and logged in, in a screen session) if there are things people want me to try.
>> This begs the questions:
>> * Is there CI for pkgbase that tries to upgrade from whatever version is immediately previous to it, before publishing it?  (I know that's what version I was running, it's the version everyone's been stuck at for weeks!)
>> * For all the debate about "pkgbase and pkg should be exactly the same", perhaps pkgbase could have an auto-bectl in it?
>> * Is there support somewhere for having a lockstep "set" of packages that one knows are in /var/db/pkg and constitute a relatively concurrent install?  (Or a list of files that I could tell pkg-static to install with a glob, out of /var/cache/pkg)
>> * With something like -CURRENT, is there any support for saying "Okay not the current-current tree, but current-minus-one"  (I guess that would be if you're working with the weekly builds, but that's not quite the same.  Your only option is "base system of the now" but it happens less frequently).
>> * And for running -CURRENT where this kind of breakage can happen, could we get a statically linked version of pkg?
>> Any questions, let me know.
>> -Dan
>> Sent from my iPhone
> 
> <OpenPGP_0xEF5BA4DCD5A9F3C0.asc>