Encrypted (GELI) root on ZFS troubles

dweimer dweimer at dweimer.net
Thu Oct 2 01:15:45 UTC 2014


On 10/01/2014 4:27 pm, Karl Denninger wrote:
> So here's the fun part of what I'm trying to do (and getting frustrated
> with)
> 
> I have set up a GPT disk with the following setup:
> 
> =>       34  625142381  da2  GPT  (298G)
>          34          6       - free -  (3.0K)
>          40       1024    1  freebsd-boot  (512K)
>        1064    4194304    2  freebsd-zfs  [bootme]  (2.0G)
>     4195368  134217728    3  freebsd-swap  (64G)
>   138413096  486729312    4  freebsd-zfs  (232G)
>   625142408          7       - free -  (3.5K)
> 
> Then on freebsd-boot I have written the bootloaders.
> 
> The "bootme" filesystem has *only* the /boot directory copied over from
> the rest of the system's root directory (that is, the kernel, 
> loadables,
> /boot/loader.conf, etc); that pool is called "zboot"
> 
> Partition 4 has the label "root0" on it, and thus shows up in /dev/gpt.
> I have initialized that with geli, set the boot option flag (that is,
> prompt on boot) and created a pool called "root" on the resulting .eli
> device and then put the system on that.  That's all ok.
> 
> Finally, I set the bootfs on that latter pool.  There is no bootfs set
> on /zboot:
> 
> # zpool get bootfs zboot
> NAME   PROPERTY  VALUE   SOURCE
> zboot  bootfs    -       default
> 
> It is set on the root pool to the proper filesystem:
> 
> # zpool get bootfs root
> NAME  PROPERTY  VALUE              SOURCE
> root  bootfs    root/R/10.1-CLEAN  local
> 
> The problem is that when the system boots geli "finds" the raw device
> (in this case /dev/da0p4), prompts for the password and attaches there
> instead of in /dev/gpt.  The gpt label is missing --- and equally bad
> the "root" pool does not appear to import at boot time either.
> 
> As a result the system tries to mount root from /zboot (even though 
> it's
> not been told to, and HAS been told where to mount off the root pool),
> but there's no init in there (or anything else other than the boot
> filesystem itself) and as a result I get an immediate panic.
> 
> If I boot off a different (working) zfs-based system the probe still
> finds the "prompt during boot" flag on that gpt partition and asks for
> the password on the device.  I can see the pool; zpool import shows it:
> 
>  pool: root
>      id: 17719633931604198170
>   state: ONLINE
>  action: The pool can be imported using its name or numeric identifier.
>  config:
> 
>         root         ONLINE
>           da2p4.eli  ONLINE
> 
> Not so good.
> 
> If I detach that the device reappears in /dev/gpt; I can then attach
> geli and import the pool in either location.  Putting the cache file
> from the previous imported state in the zboot/boot/zfs directory 
> doesn't
> help (nor does removing the cache file entirely)
> 
> More-interestingly if I reboot the cloned system with the root pool
> imported it does come back up, even though the device is the base
> (da2p4.eli) rather than in the /dev/gpt directory.
> 
> Anyone know what's going on here?  And is there a way to have geli
> attach during boot-time off the /dev/gpt directory instead of on the
> base device partition name?

On my work laptop (not turned on so I am going by memory on this), I 
have a similar setup using a USB thumb drive for the boot volume.  My 
setup is as follows and works quite well, perhaps this will help you.

Thumb Drive
da0

Disk Drive
ada0

da0 has a GPT table of
da0 GPT (8G)
1   freebsd-boot (512k) -- /dev/gpt/usbboot
2   freebsd-zfs   (8G)  -- /dev/gpt/usbzfs

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

ada0 has a GPT table of
1   freebsd-swap (8G)   -- /dev/gpt/swap
2   freebsd-zfs  (222G) -- /dev/gpt/zroot


I used geli init -b /dev/gpt/zroot
   when attached /dev/gpt/zroot.eli

swap is auto encrypted at boot using the fstab line
   /dev/gpt/swap.eli none none swap sw 0 0

I believe they devices only show up as /dev/gpt/... if the -l ... option 
is used to set a label on the partition at creation time.

2 configured zpools
usbzfs
   gpt/usbzfs

zroot
   gpt/zroot.eli

zpool set bootfs=usbzfs/boot usbzfs
zpool set bootfs=zroot/ROOT/installation zroot (not sure if this does 
anything, I just set it)

usbzfs/boot has a mountpoint of /zfsboot

loader.conf:
  zfs_load="YES"
  vfs.root.mountfrom="zfs:zroot/ROOT/install"

copied /boot to /zfsboot/boot

zpool export usbzfs

It will still boot after the zpool has been exported if the devices is 
found, just doesn't get mounted, in my case this means I can remove the 
USB thumb drive as soon as root is remounted from the geli partition, 
after entering the password without causing any issues.

I can send you the full gpt output and zpool status information tomorrow 
morning when I am back in the office on my laptop if you still need help 
getting yours working.

-- 
Thanks,
    Dean E. Weimer
    http://www.dweimer.net/


More information about the freebsd-stable mailing list