12.1p7 no longer boots after doing zpool upgrade -a

John Kennedy warlock at phouka.net
Thu Jul 9 21:20:17 UTC 2020


On Thu, Jul 09, 2020 at 08:24:54AM -0500, Kyle Evans wrote:
> On Thu, Jul 9, 2020 at 8:12 AM Guido van Rooij <guido at gvr.org> wrote:
> >
> > I did a zpool upgrade -a to enable large_dnode and spacemap_v2.
> > After that, I did:
> > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada0
> > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada1
> > and:
> > gpart bootcode -p /boot/boot1.efifat -i 1 ada0
> > gpart bootcode -p /boot/boot1.efifat -i 1 ada1
> >
> > Now the system no longer boots from either disk and drops to the efi shell.
> 
> This method of updating the ESP is no longer recommended for new 12.x
> installations -- we now more carefully construct the ESP with an
> /EFI/FreeBSD/loader.efi where loader.efi is /boot/loader.efi. You will
> want to rebuild this as such, and that may fix part of your problem.

  So, I've got a stable box:

	FreeBSD 12.1-STABLE #156 r362983+cd7a9efd329c(stable/12): Tue Jul  7 07:42:17 PDT 2020

  I've got almost the same situation, except I haven't shot myself in the foot yet.
My partitions:

	# gpart show nvd0 nvd1
	=>       40  976773088  nvd0  GPT  (466G)
	         40     409600     1  efi  (200M)
	     409640       1024     2  freebsd-boot  (512K)
	     410664        984        - free -  (492K)
	     411648   16777216     3  freebsd-swap  (8.0G)
	   17188864  959584256     4  freebsd-zfs  (458G)
	  976773120          8        - free -  (4.0K)

	=>       40  976773088  nvd1  GPT  (466G)
	         40     409600     1  efi  (200M)
	     409640       1024     2  freebsd-boot  (512K)
	     410664        984        - free -  (492K)
	     411648   16777216     3  freebsd-swap  (8.0G)
	   17188864  959584256     4  freebsd-zfs  (458G)
	  976773120          8        - free -  (4.0K)

  So this is my first UEFI system that's hit an upgrade point:

	# zpool status
	  pool: zroot
	 state: ONLINE
	status: Some supported features are not enabled on the pool. The pool can
	        still be used, but some features are unavailable.
	action: Enable all features using 'zpool upgrade'. Once this is done,
	        the pool may no longer be accessible by software that does not support
	        the features. See zpool-features(7) for details.
	  scan: scrub repaired 0 in 0 days 00:02:04 with 0 errors on Wed Jul  1 13:21:57 2020
	config:
	
	        NAME          STATE     READ WRITE CKSUM
	        zroot         ONLINE       0     0     0
	          nvd1p4.eli  ONLINE       0     0     0
	          nvd0p4.eli  ONLINE       0     0     0

  I usually seed these things off of stick and they evolve from there, but this
one is slightly interesting because the motherboard died repeatedly and I'm
guessing that the M.2 drives got reinstalled in the "wrong" order so now if I
don't manually select one of the M.2 drives, it won't reboot (acts like the
encryption password is bad, fails out into a shell where I reboot).

  In any case, I know if I upgrade, it is going to tell me to upgrade the boot
code and I'm wondering what that is these days.

  If this was a Pre-UEFI, I would be expecting to do (only) something like:

	gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nvd0
	gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nvd1

  IE, might hit either drive, update the freebsd-boot partition by # on both,
and special note that the # is now one higher because of the EFI partition.

  The FreeBSD wiki still has the "old" way of doing it here...

	https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot

  ... but your newer way of doing it here:

	https://wiki.freebsd.org/UEFI

  So the newer way of doing it would be something like this, except I'd by updating
the file rather than looking at it's checksum/version?

	strings -a < /boot/loader.efi | grep -A1 'EFI loader'
		FreeBSD/amd64 EFI loader, Revision 1.1
		(Tue Jul  7 07:33:24 PDT 2020 warlock at ouroboros.phouka.net)

	mount -t msdosfs /dev/nvd0p1 /mnt
	md5 -r /boot/loader.efi /mnt/efi/boot/BOOTx64.efi
		9f64a96f3170e3327ca7dad1d574d852 /boot/loader.efi
		cbff58cc8ee79ce7342f5be23ad14ba3 /mnt/efi/boot/BOOTx64.efi
	strings -a < /mnt/efi/boot/BOOTx64.efi | grep -A1 'EFI loader'
		FreeBSD/amd64 EFI loader, Revision 1.1
		|/-\
	umount -v /mnt

	mount -t msdosfs /dev/nvd1p1 /mnt
	md5 -r /mnt/efi/boot/BOOTx64.efi
		cbff58cc8ee79ce7342f5be23ad14ba3 /mnt/efi/boot/BOOTx64.efi
	umount -v /mnt

  At least some of the checksum difference is because WITHOUT_REPRODUCIBLE_BUILD=YES.

  (Sorry for extra details, but I know when I was researching this, I ran
   across a *lot* of examples where we didn't have a GPT dump and people
   were just using partition #s and if you stomp on the wrong partition
   you're going to have a bad day, so leaving some specific examples here.)

  So one recipe doesn't even seem to make the freebsd-boot partition, so is it
optional for a pure UEFI boot?  Should we always gpart-bootcode it if it exists
and upgrade BOOTx64.efi when the EFI partition exists, or is there a few more
wrinkles we need to worry about?


  In any case, is it a logical theory that my possible boot-order problem
is because drive order got swapped and maybe one wasn't properly updated?
They seem to be the same:

	# md5 /dev/nvd[01]p2
		MD5 (/dev/nvd0p2) = 2ded438a2c97ea551446cc2d1d3b498e
		MD5 (/dev/nvd1p2) = 2ded438a2c97ea551446cc2d1d3b498e

  Ideally I'd like to have not boot through the UEFI boot menu every time.

  I'm not sure why the drive order seems to matter right now.



More information about the freebsd-stable mailing list