13.3-RELEASE upgrade woes, help requested

From: Scott Bennett <bennett_at_sdf.org>
Date: Wed, 04 Sep 2024 09:11:37 UTC
     First, I'll describe the two very old computers involved.  One is a laptop
with an AMD A6 CPU and 8 GB of RAM and integrated graphics.  The peripheral
storage is a SanDisk SATA 476.9 GB SSD limited to SATA-2 speeds by the very old
SATA controller on a PCIe v2 motherboard.  This is the machine with which I am
currently ssh'ed into a system at sdf.org, where I deal with email.
     The other computer is a Dell tower with a Core2 Extreme (QX9650) CPU with
8 GB of RAM and a now unsupported Radeon graphics card.  The peripheral storage
devices are two ~.9 TB HDDs and six ~1.86 TB or larger capacity HDDs.  The two
smaller drives and two of the larger drives are attached to the four available
internal SATA-2 ports, and three of the other drives are attached to USB 3.0
ports on add-in cards on the motherboard.  The last drive is attached to a
port on a Jmicron ESATA port at SATA-1 speed.  The two smallest drives are the
boot devices and are each partitioned with the boot loader's UFS2 partition, a 
a partition containing one component of a two-way ZFS mirror that is a boot
partition with a pool name of "system".  There is also a small partition on
each for the crash dump area on one and /var/crash (UFS2) on the other.  There
is also a 2 GB partition on each drive for a GEOM mirror that supports a UFS2
partition for an application.  Lastly, most of the remaining space on the
two small drives are a two-way ZFS mirror containing /usr/home and potentially
other file systems.  That pool's name is "local".  The four remaining pools
have several things, including the two remaining pools ("rz7A", a raidz2 pool
with 6 components totalling ~10.4 TB, and "zmisc", comprising two mirrored
vdevs and totalling 99 GB) and three small GEOM mirrors of varying sizes
GEOM-concatenated together to hold a UFS2 file system for a work area for
ccache trees and WRKDIRPREFIX for portmaster(8).  "system", "local", and "rz7A"
are all on GELI-encrypted partitions.  "zmisc" is not encrypted.
     Note that what follows is to the best of my memory, but with the caveat
that I may be recalling incorrectly which releases and steps occurred in which
order for the early history, but if I am, that is really of not much importance
here.  Also, neither machine is EFI-capable due to their ages.
     As of many moons ago, I *think* the tower, which is my primary machine,
was running 12.2-RELEASE-p[x] (i.e., I don't remember the patch level).  The
laptop, my experimental and now emergency machine, had some patch level of
13.1-RELEASE on it, IIRC.  For many years I have been uprading FreeBSD from
source, but decided to try the freebsd-update(8) process when 13.2-RELEASE was
released.  I did that and quickly discovered that it had completely undone much
or all of my OS configuration, especially the network configuration, that I had
tailored to my needs, so I wasted seemingly endless hours over weeks repairing
all that into some semblance of usable form.  Lesson learned.  Meanwhile I had
proceeded with source upgrade of the tower to 12.3-RELEASE-p[x].
     When 13.3-RELEASE became available, I first did a source upgrade on the
laptop.  Running "make installworld" rendered the laptop unbootable.  I don't
any longer recall all the things I tried to salvage that system, but eventually
I removed the SSD from it and loaded it into a USB 3.0 docking station and
attached it to the tower and replicated its entire pool ("sysroot") into my
largest pool attached to the tower.  Then I reinstalled the SSD into the 
laptop.  After downloading the 13.3-RELEASE image onto the tower and writing
it to a thumb drive, I booted that on the laptop and installed 13.3-RELEASE
from scratch.  After some network configuration I was then able to rescue some
critical files from the backup on the tower in order to make it sort of usable.
     That experience delayed my upgrading the tower to 13.3-RELEASE-p1 for
several months, although I had compiled it on the tower and had it ready to
install, but first I had upgraded the tower to 12.4-RELEASE-p2.  Finally I
dared to try it a day and several hours ago.  I first created a boot
environment to preserve the current system and also made a snapshot of all ZFS
file system in order to have a potential rollback point before beginning the
installkernel step.
     I rebooted the tower after the "make installkernel KERNCONF=hellas" step.
That worked, but I soon discovered that the only pool that it opened was
"system", i.e., the boot pool.  I tried importing the other three pools and
was informed that each one had been in using by another system(!), so I would
have to use "zpool import -f {poolname}".  I did that, completed the etcupdate
steps, and then did a "shutdown -r now".  After the boot loader asks me for
the GELI passphrase for the boot pool, I get the following.

Calculating GELI Decryption Key for disk0p2: 1563240 iterations...

BTX loader 1.00  BTX version is 1.02

After those two lines the blinking cursor jumps up three lines and moves to the
beginning of the line--not sure which happens first because it's too fast.
After a delay of several seconds, it jumps two lines down and repeats the
delay and downward jump two or three times, then jumps to two lines above the
bottom line of the screen.  After a lengthy delay it jumps to the bottom line.
After a much longer delay the cursor jumps three spaces to the right and never
moves after that point and is unresponsive, although CTL-Alt-Delete can still
cause a BIOS reset and eventual attempt at reboot.  Once again a 13.3-RELEASE
installworld has rendered a system unbootable. :-(
     If someone can kindly explain to me what may have gone wrong and/or what
I need to do to fix it, I would be very grateful.  That USB 3.0 storage device
docking station can be moved and connected to the laptop to attempt a repair,
provided I have an idea how to do the repair.  I really, really do *not* want
to begin all over with an unconfigured installation onto a drive for the
following reasons.  The braindead ZFS installer will overwrite my painstakingly
set up partitions and file systems.  It was a huge nuisance to get that storage
configuration the way I want it, and I do *not* want to go through all that or
anything similar again.  It required having a second device properly
partitioned, ZFS pools and GEOM mirror configured, etc. to do it.  In order to
recover my home directory and other worthy data, I would need to have a blank
drive available that I do not now have and many hours for the replications back
and forth between the three drives that would be involved, and the process
would not be identical to the original process.  In short, this would be a very
last resort, a truly unpleasant and lousy option.
     Of course, while the tower is down much work it normally does around the
clock is stalled. :-(  Thus I'd really like to get the erroneous data on its
boot drives repaired soon.
     Thanks in advance to anyone who can help!  Please Cc: me on any replies
because I am subscribed to this list as a digest, which can often delay replies
reaching me for a day or occasionally longer.

					Scott Bennett