Encrypted (GELI) root on ZFS troubles
Karl Denninger
karl at denninger.net
Thu Oct 2 04:51:15 UTC 2014
On 10/1/2014 8:15 PM, dweimer wrote:
> On 10/01/2014 4:27 pm, Karl Denninger wrote:
>> So here's the fun part of what I'm trying to do (and getting frustrated
>> with)
>>
>> I have set up a GPT disk with the following setup:
>>
>> => 34 625142381 da2 GPT (298G)
>> 34 6 - free - (3.0K)
>> 40 1024 1 freebsd-boot (512K)
>> 1064 4194304 2 freebsd-zfs [bootme] (2.0G)
>> 4195368 134217728 3 freebsd-swap (64G)
>> 138413096 486729312 4 freebsd-zfs (232G)
>> 625142408 7 - free - (3.5K)
>>
>> Then on freebsd-boot I have written the bootloaders.
>>
>> The "bootme" filesystem has *only* the /boot directory copied over from
>> the rest of the system's root directory (that is, the kernel, loadables,
>> /boot/loader.conf, etc); that pool is called "zboot"
>>
>> Partition 4 has the label "root0" on it, and thus shows up in /dev/gpt.
>> I have initialized that with geli, set the boot option flag (that is,
>> prompt on boot) and created a pool called "root" on the resulting .eli
>> device and then put the system on that. That's all ok.
>>
>> Finally, I set the bootfs on that latter pool. There is no bootfs set
>> on /zboot:
>>
>> # zpool get bootfs zboot
>> NAME PROPERTY VALUE SOURCE
>> zboot bootfs - default
>>
>> It is set on the root pool to the proper filesystem:
>>
>> # zpool get bootfs root
>> NAME PROPERTY VALUE SOURCE
>> root bootfs root/R/10.1-CLEAN local
>>
>> The problem is that when the system boots geli "finds" the raw device
>> (in this case /dev/da0p4), prompts for the password and attaches there
>> instead of in /dev/gpt. The gpt label is missing --- and equally bad
>> the "root" pool does not appear to import at boot time either.
>>
>> As a result the system tries to mount root from /zboot (even though it's
>> not been told to, and HAS been told where to mount off the root pool),
>> but there's no init in there (or anything else other than the boot
>> filesystem itself) and as a result I get an immediate panic.
>>
>> If I boot off a different (working) zfs-based system the probe still
>> finds the "prompt during boot" flag on that gpt partition and asks for
>> the password on the device. I can see the pool; zpool import shows it:
>>
>> pool: root
>> id: 17719633931604198170
>> state: ONLINE
>> action: The pool can be imported using its name or numeric identifier.
>> config:
>>
>> root ONLINE
>> da2p4.eli ONLINE
>>
>> Not so good.
>>
>> If I detach that the device reappears in /dev/gpt; I can then attach
>> geli and import the pool in either location. Putting the cache file
>> from the previous imported state in the zboot/boot/zfs directory doesn't
>> help (nor does removing the cache file entirely)
>>
>> More-interestingly if I reboot the cloned system with the root pool
>> imported it does come back up, even though the device is the base
>> (da2p4.eli) rather than in the /dev/gpt directory.
>>
>> Anyone know what's going on here? And is there a way to have geli
>> attach during boot-time off the /dev/gpt directory instead of on the
>> base device partition name?
>
> On my work laptop (not turned on so I am going by memory on this), I
> have a similar setup using a USB thumb drive for the boot volume. My
> setup is as follows and works quite well, perhaps this will help you.
>
> Thumb Drive
> da0
>
> Disk Drive
> ada0
>
> da0 has a GPT table of
> da0 GPT (8G)
> 1 freebsd-boot (512k) -- /dev/gpt/usbboot
> 2 freebsd-zfs (8G) -- /dev/gpt/usbzfs
>
> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
>
> ada0 has a GPT table of
> 1 freebsd-swap (8G) -- /dev/gpt/swap
> 2 freebsd-zfs (222G) -- /dev/gpt/zroot
>
>
> I used geli init -b /dev/gpt/zroot
> when attached /dev/gpt/zroot.eli
>
> swap is auto encrypted at boot using the fstab line
> /dev/gpt/swap.eli none none swap sw 0 0
>
> I believe they devices only show up as /dev/gpt/... if the -l ...
> option is used to set a label on the partition at creation time.
>
> 2 configured zpools
> usbzfs
> gpt/usbzfs
>
> zroot
> gpt/zroot.eli
>
> zpool set bootfs=usbzfs/boot usbzfs
> zpool set bootfs=zroot/ROOT/installation zroot (not sure if this does
> anything, I just set it)
>
> usbzfs/boot has a mountpoint of /zfsboot
>
> loader.conf:
> zfs_load="YES"
> vfs.root.mountfrom="zfs:zroot/ROOT/install"
>
> copied /boot to /zfsboot/boot
>
> zpool export usbzfs
>
> It will still boot after the zpool has been exported if the devices is
> found, just doesn't get mounted, in my case this means I can remove
> the USB thumb drive as soon as root is remounted from the geli
> partition, after entering the password without causing any issues.
>
> I can send you the full gpt output and zpool status information
> tomorrow morning when I am back in the office on my laptop if you
> still need help getting yours working.
>
I got it working but.....
1. Having the kernel able to cross pools and examine the bootfs property
would be fairly useful. I use beadm, but of course there are lots of
risks associated with it and this sort of setup; if you forget to update
the unencrypted boot area you could find yourself in pretty-serious
trouble with an unbootable system -- if your emergency media is also out
of date vis-a-vis zfs flags you've got trouble.
2. Not having the system look at the /dev/gpt entries (yes, I use "-l"
when I create the partitions) is troublesome. Specifically, it's
trouble when you have a lot of drives and an adapter that doesn't play
nice with coherent unit numbering related to the cable point the disk is
on -- which is true of most of the mps-driven adapters (and especially
those with a SAS expander or three hooked to them.) If you've never
pulled the wrong disk from a running machine by thinking the numbering
is different than it actually is then this probably doesn't mean much to
you :-) Not being able to be SURE which disk failed so you can make
sure you yank the right one is a quite serious annoyance.
--
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2711 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20141001/59f47a90/attachment.bin>
More information about the freebsd-stable
mailing list