12.1p7 no longer boots after doing zpool upgrade -a

[Kyle Evans]

On Thu, Jul 9, 2020 at 4:20 PM John Kennedy <warlock at phouka.net> wrote:
>
> On Thu, Jul 09, 2020 at 08:24:54AM -0500, Kyle Evans wrote:
> > On Thu, Jul 9, 2020 at 8:12 AM Guido van Rooij <guido at gvr.org> wrote:
> > >
> > > I did a zpool upgrade -a to enable large_dnode and spacemap_v2.
> > > After that, I did:
> > > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada0
> > > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada1
> > > and:
> > > gpart bootcode -p /boot/boot1.efifat -i 1 ada0
> > > gpart bootcode -p /boot/boot1.efifat -i 1 ada1
> > >
> > > Now the system no longer boots from either disk and drops to the efi shell.
> >
> > This method of updating the ESP is no longer recommended for new 12.x
> > installations -- we now more carefully construct the ESP with an
> > /EFI/FreeBSD/loader.efi where loader.efi is /boot/loader.efi. You will
> > want to rebuild this as such, and that may fix part of your problem.
>
>   So, I've got a stable box:
>
>         FreeBSD 12.1-STABLE #156 r362983+cd7a9efd329c(stable/12): Tue Jul  7 07:42:17 PDT 2020
>
>   I've got almost the same situation, except I haven't shot myself in the foot yet.
> My partitions:
>
>         # gpart show nvd0 nvd1
>         =>       40  976773088  nvd0  GPT  (466G)
>                  40     409600     1  efi  (200M)
>              409640       1024     2  freebsd-boot  (512K)
>              410664        984        - free -  (492K)
>              411648   16777216     3  freebsd-swap  (8.0G)
>            17188864  959584256     4  freebsd-zfs  (458G)
>           976773120          8        - free -  (4.0K)
>
>         =>       40  976773088  nvd1  GPT  (466G)
>                  40     409600     1  efi  (200M)
>              409640       1024     2  freebsd-boot  (512K)
>              410664        984        - free -  (492K)
>              411648   16777216     3  freebsd-swap  (8.0G)
>            17188864  959584256     4  freebsd-zfs  (458G)
>           976773120          8        - free -  (4.0K)
>
> [... snip ...]
>
>   If this was a Pre-UEFI, I would be expecting to do (only) something like:
>
>         gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nvd0
>         gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nvd1
>
>   IE, might hit either drive, update the freebsd-boot partition by # on both,
> and special note that the # is now one higher because of the EFI partition.
>
>   The FreeBSD wiki still has the "old" way of doing it here...
>
>         https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot
>
>   ... but your newer way of doing it here:
>
>         https://wiki.freebsd.org/UEFI
>
>   So the newer way of doing it would be something like this, except I'd by updating
> the file rather than looking at it's checksum/version?
>
>         strings -a < /boot/loader.efi | grep -A1 'EFI loader'
>                 FreeBSD/amd64 EFI loader, Revision 1.1
>                 (Tue Jul  7 07:33:24 PDT 2020 warlock at ouroboros.phouka.net)
>
>         mount -t msdosfs /dev/nvd0p1 /mnt
>         md5 -r /boot/loader.efi /mnt/efi/boot/BOOTx64.efi
>                 9f64a96f3170e3327ca7dad1d574d852 /boot/loader.efi
>                 cbff58cc8ee79ce7342f5be23ad14ba3 /mnt/efi/boot/BOOTx64.efi
>         strings -a < /mnt/efi/boot/BOOTx64.efi | grep -A1 'EFI loader'
>                 FreeBSD/amd64 EFI loader, Revision 1.1
>                 |/-\
>         umount -v /mnt
>
>         mount -t msdosfs /dev/nvd1p1 /mnt
>         md5 -r /mnt/efi/boot/BOOTx64.efi
>                 cbff58cc8ee79ce7342f5be23ad14ba3 /mnt/efi/boot/BOOTx64.efi
>         umount -v /mnt
>
>   At least some of the checksum difference is because WITHOUT_REPRODUCIBLE_BUILD=YES.
>

Yes, this looks correct to me.

>   (Sorry for extra details, but I know when I was researching this, I ran
>    across a *lot* of examples where we didn't have a GPT dump and people
>    were just using partition #s and if you stomp on the wrong partition
>    you're going to have a bad day, so leaving some specific examples here.)
>
>   So one recipe doesn't even seem to make the freebsd-boot partition, so is it
> optional for a pure UEFI boot?  Should we always gpart-bootcode it if it exists
> and upgrade BOOTx64.efi when the EFI partition exists, or is there a few more
> wrinkles we need to worry about?
>

Correct, freebsd-boot is not needed for a pure UEFI boot. I wouldn't
necessarily bother updating the freebsd-boot partition unless you
suspect you may need to switch back to legacy boot at some point; UEFI
is now rock-solid on all of my systems, so I've personally found no
such need and on many of them I've removed the old freebsd-boot bits.

If you've got an ESP, you should update that manually. If you want to
maintain the option of legacy boot, you should use gpart-bootcode for
that but don't use it on the ESP with boot1.efifat.

>   In any case, is it a logical theory that my possible boot-order problem
> is because drive order got swapped and maybe one wasn't properly updated?
> They seem to be the same:
>
>         # md5 /dev/nvd[01]p2
>                 MD5 (/dev/nvd0p2) = 2ded438a2c97ea551446cc2d1d3b498e
>                 MD5 (/dev/nvd1p2) = 2ded438a2c97ea551446cc2d1d3b498e
>
>   Ideally I'd like to have not boot through the UEFI boot menu every time.
>
>   I'm not sure why the drive order seems to matter right now.
>

When you get booted back to the UEFI menu, is it a specific drive that
you must select or do both equally work from that point?

Thanks,

Kyle Evans