12.0->12.1 and beadm/bectl issues

[Jon Tibble]

Hi,

After upgrading from 12.0-RELEASE-p11 to 12.1-RELEASE I was having some 
issues with kld_load and linux support which, after searching [1], 
seemed due to a missing /boot folder after the upgrade.
This was fixed with 'ln -s /bootpool/boot /boot'.

Then yesterday when I was trying to switch from quarterly packages to 
latest I wanted to use a new boot environment and so went through the 
beadm create and beadm activate but it wouldn't activate with a 
zpool.cache cp message and it left the new BE mounted under /tmp.
After umounting and destroying I repeated the process with bectl and it 
worked fine, however, upon reboot I was not in the new BE but the same 
BE and the new one was still marked as activated for use next boot.

So firstly: are the be* issues related to the earlier upgrade fix?

Secondly: shouldn't beadm and bectl behave the same?

Thirdly: how can I properly activate and boot to a new BE?

Below is the command output of the beadm/bectl process described above. 
If there's any more information I can provide please let me know.

Thanks,
Jon

[1] 
https://forums.freebsd.org/threads/cannot-identify-running-kernel-after-upgrading-to-freebsd-12.68772/


This is a two disk mirrored zpool on GELI with encrypted swap as 
configured out of the box by the 12.0 installer.

root at prometheus:~ # uname -a
FreeBSD prometheus 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 GENERIC  amd64
root at prometheus:~ # beadm list
BE                        Active Mountpoint  Space Created
12_0-RELEASE-p11          -      -            1.1G 2019-10-29 21:33
12_1-RELEASE-p1-quarterly NR     /           32.7G 2019-11-05 22:24
root at prometheus:~ # beadm create test
Created successfully
root at prometheus:~ # beadm list
BE                        Active Mountpoint  Space Created
12_0-RELEASE-p11          -      -            1.1G 2019-10-29 21:33
12_1-RELEASE-p1-quarterly NR     /           32.7G 2019-11-05 22:24
test                      -      -            8.0K 2019-11-20 13:24
root at prometheus:~ # beadm activate test
cp: /tmp/BE-test.pJtR9Rs6/boot/zfs/zpool.cache and /boot/zfs/zpool.cache 
are identical (not copied).
root at prometheus:~ # beadm list
BE                        Active Mountpoint             Space Created
12_0-RELEASE-p11          -      -                       1.1G 2019-10-29 
21:33
12_1-RELEASE-p1-quarterly NR     /                      32.7G 2019-11-05 
22:24
test                      -      /tmp/BE-test.pJtR9Rs6 136.0K 2019-11-20 
13:24
root at prometheus:~ # beadm umount test
Unmounted successfully
root at prometheus:~ # beadm list
BE                        Active Mountpoint  Space Created
12_0-RELEASE-p11          -      -            1.1G 2019-10-29 21:33
12_1-RELEASE-p1-quarterly NR     /           32.7G 2019-11-05 22:24
test                      -      -          136.0K 2019-11-20 13:24
root at prometheus:~ # beadm destroy test
Are you sure you want to destroy 'test'?
This action cannot be undone (y/[n]): y
Destroyed successfully
root at prometheus:~ # beadm list
BE                        Active Mountpoint  Space Created
12_0-RELEASE-p11          -      -            1.1G 2019-10-29 21:33
12_1-RELEASE-p1-quarterly NR     /           32.7G 2019-11-05 22:24
root at prometheus:~ # bectl list
BE                        Active Mountpoint Space Created
12_0-RELEASE-p11          -      -          1.14G 2019-10-29 21:33
12_1-RELEASE-p1-quarterly NR     /          32.7G 2019-11-05 22:24
root at prometheus:~ # bectl create test
root at prometheus:~ # bectl list
BE                        Active Mountpoint Space Created
12_0-RELEASE-p11          -      -          1.14G 2019-10-29 21:33
12_1-RELEASE-p1-quarterly NR     /          32.7G 2019-11-05 22:24
test                      -      -          8K    2019-11-20 13:25
root at prometheus:~ # bectl activate test
successfully activated boot environment test
root at prometheus:~ # bectl list
BE                        Active Mountpoint Space Created
12_0-RELEASE-p11          -      -          1.14G 2019-10-29 21:33
12_1-RELEASE-p1-quarterly N      /          8K    2019-11-05 22:24
test                      R      -          32.7G 2019-11-20 13:25
root at prometheus:~ # beadm list
BE                        Active Mountpoint  Space Created
12_0-RELEASE-p11          -      -            1.1G 2019-10-29 21:33
12_1-RELEASE-p1-quarterly N      /            8.0K 2019-11-05 22:24
test                      R      -           32.7G 2019-11-20 13:25
root at prometheus:~ #

Following a reboot I'll still be running in 12_1-RELEASE-p1-quarterly 
and test will still be marked R.