Re: Booting in bhyve always sets currdev to ZFS

On Tue, Jul 2, 2024 at 7:30 AM Rodney W. Grimes
<freebsd-rwg@gndrsh.dnsmgr.net> wrote:
>
> > On Sun, Jun 30, 2024 at 8:23?AM Rodney W. Grimes
> > <freebsd-rwg@gndrsh.dnsmgr.net> wrote:
> > >
> > > > On Fri, Jun 28, 2024 at 6:26?PM Rick Macklem <rick.macklem@gmail.com> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I've installed FreeBSD current in a bhvye instance. Everything
> > > > > went ok, with UFS as a root partition.
> > > > > Then I created a zpool in another partition...
> > > > > - Now, every time I boot it I have to
> > > > >   OK set currdev=disk0s1a:
> > > > > to get it to boot.
> > > > >
> > > > > What is the trick to keep ZFS from messing up the boot variables?
> > > > I've been poking around, but haven't learned much.
> > > > I think it is userboot (although there are so many boot programs in /boot,
> > > > I am not 100% sure?) that sets currdev=zfs:example: since it sees there is ZFS
> > > > in a partition on the drive. It is not the boot partition and doesn't have
> > > > any boot stuff in it.
> > > >
> > > > When I look at userboot, it appears that it always sets userboot_zfs_found
> > > > to 1 whenever userboot_zfs_probe() is called, given that there is a ZFS
> > > > partition with a pool on it. This makes extract_currdev get set to the
> > > > ZFS stuff,
> > > > assuming I am reading the code correctly.
> > > > What I do not understand is why I have not seen this before?
> > > > (Was the a change to building it with USERBOOT_ZFS_SUPPORT done?)
> > > >
> > > > It seems that userboot_zfs_probe() should check for boot files on the volume
> > > > and not just that a pool exists on the partition, maybe?
> > > >
> > > > Anyhow, manually setting currdev=disk0s1a: gets around the problem.
> > >
> > > You should be able to set that in your ufs partition /boot/loader.conf
> > > file and at least then your not having to drop to the OK prompt and
> > > manually entering this.
> > Doesn't help, because it is not reading files from the boot ufs file system.
> > (I could copy all the boot stuff over into the ZFS file system, but that kinda
> > defeats the purpose of having UFS as the boot fs.)
>
> It had to of loaded userboot from the ufs, right?
I assume it is reading some boot file from UFS. Way back when I thought a
primary boot was read from 8K at the begining of the partition, but that's
how out of date I on this.
-> Put another way, I don't know if "userboot" is the one I am seeing try
   to use the ZFS partition.

>
> What type of disk is this?  gpt or mbr?
It's old MBR. Although I'll assume bhyve knows other stuff, most of my old
hardware only does MBR and not EFI, so I'll still in the MBR habit.

>
> Can you show the output of gpart show and gpart list?
root@nfsv4-foo:~ # gpart show
=>      63  67108801  vtbd0  MBR  (32G)
        63         1         - free -  (512B)
        64  67108800      1  freebsd  [active]  (32G)

=>       0  67108800  vtbd0s1  BSD  (32G)
         0  33554432        1  freebsd-ufs  (16G)
  33554432  12582912        2  freebsd-swap  (6.0G)
  46137344  20971456        4  freebsd-zfs  (10G)

root@nfsv4-foo:~ # gpart list
Geom name: vtbd0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 67108863
first: 63
entries: 4
scheme: MBR
Providers:
1. Name: vtbd0s1
   Mediasize: 34359705600 (32G)
   Sectorsize: 512
   Stripesize: 32768
   Stripeoffset: 0
   Mode: r3w3e5
   efimedia: HD(1,MBR,0x90909090,0x40,0x3ffffc0)
   attrib: active
   rawtype: 165
   length: 34359705600
   offset: 32768
   type: freebsd
   index: 1
   end: 67108863
   start: 64
Consumers:
1. Name: vtbd0
   Mediasize: 34359738368 (32G)
   Sectorsize: 512
   Stripesize: 32768
   Stripeoffset: 0
   Mode: r3w3e8

Geom name: vtbd0s1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 67108799
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: vtbd0s1a
   Mediasize: 17179869184 (16G)
   Sectorsize: 512
   Stripesize: 32768
   Stripeoffset: 0
   Mode: r1w1e1
   rawtype: 7
   length: 17179869184
   offset: 0
   type: freebsd-ufs
   index: 1
   end: 33554431
   start: 0
2. Name: vtbd0s1b
   Mediasize: 6442450944 (6.0G)
   Sectorsize: 512
   Stripesize: 32768
   Stripeoffset: 0
   Mode: r1w1e0
   rawtype: 1
   length: 6442450944
   offset: 17179869184
   type: freebsd-swap
   index: 2
   end: 46137343
   start: 33554432
3. Name: vtbd0s1d
   Mediasize: 10737385472 (10G)
   Sectorsize: 512
   Stripesize: 32768
   Stripeoffset: 0
   Mode: r1w1e1
   rawtype: 27
   length: 10737385472
   offset: 23622320128
   type: freebsd-zfs
   index: 4
   end: 67108799
   start: 46137344
Consumers:
1. Name: vtbd0s1
   Mediasize: 34359705600 (32G)
   Sectorsize: 512
   Stripesize: 32768
   Stripeoffset: 0
   Mode: r3w3e5

>
> >
> > For example, it always fails with:
> > ERROR: cannot open /boot/lua/loader.lua: no such file or directory.
> >
> > When I look in stand/efi/loader/main.c, there is a function that checks
> > for boot files called sanity_check_currdev(). However, userboot does not
> > have a similar function and just seems to assume that the ZFS partition
> > in the bootable one, if it finds one.
>
> That sounds like a wrong assumption by the code.
>
> > --> I am wondering if sanity_check_currdev() shoud be added to userboot
> >       to handle this case?
>
> Warner??
>
> > rick
> >
> > >
> > > There is much magic in boot code that bites, and not just in FreeBSD.
> > > Often assumptions are made that "this" boot code, and "this" OS are
> > > the only things on the disk(s).
> > >
> > > The fact that Linux uses a "apple-zfs" uuid, and FreeBSD a "FreeBSD-zfs"
> > > drove me nuts for 2 days until I found this little tid bit.
> > >
> > > Zpool import doesnt care, but the boot code does!
> > >
> > > --
> > > Rod Grimes                                                 rgrimes@freebsd.org
> --
> Rod Grimes                                                 rgrimes@freebsd.org