Re: FreeBSD, boot environments and /dev

Hi Wes,

finally I get round to responding (below) ...

On Fri, May 6, 2022 at 12:15 AM Wes Maag <jwmaag@gmail.com> wrote:
>
>
>
> On Thu, May 5, 2022 at 4:10 PM Michael Schuster <michaelsprivate@gmail.com> wrote:
>>
>> Hi all,
>>
>> while still working (slowly) on an answer to my own question on the
>> right workflow to keep current up to date reliably with boot
>> environments, I noticed that after creating and mounting a new BE,
>> that new BE's /dev (eg /mnt/dev) is very sparsely populated:
>>
>> [22:44:45: ~ 0] $ ll /mnt/dev
>> total 20
>> 9 dr-xr-xr-x   4 root  wheel   7 Apr 18 00:35 .
>> 9 drwxr-xr-x  23 root  wheel  30 May  5 22:44 ..
>> 1 drwxr-xr-x   2 root  wheel   2 Apr 18 00:35 fd
>> 1 lrwxr-xr-x   1 root  wheel  12 Apr 17 20:41 log -> /var/run/log
>> 1 -rw-r--r--   1 root  wheel  63 May  3 22:19 null
>> 1 -rw-r--r--   1 root  wheel   1 May  3 22:18 null.bak
>> 1 drwxr-xr-x   2 root  wheel   2 Apr 18 00:35 shm
>> [22:44:48: ~ 0] $
>>
>> (no matter whether I use "beadm create" or "bectl create -r")
>>
>> I can then mount devfs:
>> # mount -t devfs devfs /mnt/dev
>>
>> and it will look (and work) as expected (eg for "pkg -c /mnt update"
>> (*)) ... as long as the BE is mounted.
>> When I try to boot from (into?) that BE though, it fails, and on the
>> console I can see messages about missing /dev/... entries. I sneaked a
>> peek into beinstall.sh, found no inspiration there, I'm afraid.
>>
>> I believe I noticed this apparent requirement to mount devfs
>> separately not so long ago ... definitely this year, while I've been
>> experimenting with boot environments for quite a bit longer. Was I
>> just lucky all along, or did something change ... and, more
>> importantly, how do I make my boot environments bootable again?
>>
>> TIA for hints, pointers, advice
>> Michael
>>
>> *) I first noticed something was amiss when "pkg -c /mnt ... "
>> complained about /dev/null missing, IIRC ...
>> --
>> Michael Schuster
>> http://recursiveramblings.wordpress.com/
>> recursion, n: see 'recursion'
>>
>
> getting /dev/null errors with pkg -c makes sense. I wouldn't expect /dev to contain anything after a create.

as I wrote, I *thought* that worked automatically ... apparently not.

> Why it is not populated on boot seems odd to me though... It's possible, if you created a deep boot environment, you created a pool/ROOT/dev dataset
> and it is overwriting the initial /dev from boot (I have not tested this theory). Tough to say without more information about the BE layout, and maybe a look at fstab.

(after beadm create and beadm mount):
$ zfs list -tall
[...]
tank/ROOT/BE_20220507_171901_adm
    184K   397G  17.9G  /
tank/ROOT/BE_20220507_171901_adm/ports
      8K   397G  1.92G  /usr/ports
tank/ROOT/BE_20220507_171901_adm/src
      8K   397G  2.57G  /usr/src

$ mount | grep dev
devfs on /dev (devfs)
/dev/nvd0p1 on /boot/efi (msdosfs, local)
devfs on /compat/linux/dev (devfs)
fdescfs on /compat/linux/dev/fd (fdescfs)
tmpfs on /compat/linux/dev/shm (tmpfs, local)

$ cat /etc/fstab
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/nvd0p1             /boot/efi       msdosfs rw              2       2
/dev/nvd0p3             none    swap    sw              0       0
proc                    /proc   procfs  rw              0               0

> FWIW I typically run "pkg -r /tmp/be_mount.XXXX" after mounting with "bectl mount" which should give you the same effect as -c without the rooted devfs requirement

good point ... until now, "pkg -c" worked as expected, I never looked
further ;-) I'll keep it in mind!

I expected (perhaps naively) that with -c, pkg would be using the
"new" stuff I'd installed in the previous step (see below), not the
currently running bits ... Any thoughts on that?

One more data point, perhaps that's relevant:
I've been updating these BEs using a sequence of installing freshly
built current into /mnt (using "DESTDIR=/mnt"), following by a few
ports (drm-devel, primarily), and then doing pkg update/upgrade.

thx
Michael

-- 
Michael Schuster
http://recursiveramblings.wordpress.com/
recursion, n: see 'recursion'