9-stable from i386 to amd64

[Dan Nelson]

In the last episode (Feb 10), Randy Bush said:
> is there a recipe for moving from i386 to amd64?
> 
> on a very remote system, i made the migration from 7.4 to 8.2 to 9.0, all
> 32-bit.  it was done with repeated
> 
>     make buildworld
>     make kernel.new [0]
>     nextboot -k kernel.new
>     reboot
>     make installworld
>     etc
> 
>     [0] - well, there were some mv(1)s in there :)
> 
> so after it was happy with 9.0 i386, i went to move to amd64 with
> 
>     make buildworld TARGET=amd64
>     make kernel TARGET=amd64 DESTDIR=kernel.new [0]
>     nextboot -k kernel.new
>     reboot
> 
> it did not come back from the reboot, and required a manual reset.  i have
> no console access to the machine, not my choice.
> 
> clue bat please.

You probably got bit by a mismatched /libexec/ld-elf.so. The kernel expects
that to be the "native" version, and on a 64-bit kernel it also expects a
ld-elf32.so to be the "compat" 32-bit version.  When you rebooted onto the
64-bit kernel, it couldn't find /libexec/ld-elf32.so to run any of the
32-bit binaries on the system.  My guess is that your reboot attempt died in
/sbin/init, prompting for a path to /bin/sh.  If you compiled with a static
/bin/sh for performance, it probably died very early in /etc/rc.

I think copying ld-elf.so over to ld-elf32.so might have been all you needed
to boot, but that would end up with a 64-bit kernel running a true 32-bit
userland with all the libraries in the "wrong" place, and your
"installworld" step would replace them with their 64-bit equivalents and
your install would die halfway through, leaving you with a large mess to
clean up.

The cleanest upgrade path is to prepare your 32-bit root to be bootable by
both 32- and 64-bit kernels: copy the ld-elf32.so that was built during your
buildworld over to /libexec/ld-elf32.so, and also make copies of
/lib and /usr/lib to /lib32 and /usr/lib32 respectively.  That way when you
reboot to a 64-bit kernel, your 32-bit executables will be running
"correctly" out of compat32 paths and your installworld should succeed.

When I did all this on a local system, I made judicious use of ZFS snapshots
and clones, preserving a bootable clone of my original system plus
intermediate versions all the way until I was happy with the result.  I've
never done it completely remotely, but if you do a trial run or two on a
local machine or VM, you should be able to it confidently remotely.

-- 
	Dan Nelson
	dnelson at allantgroup.com