eli encrypted providers for zfs raidz1

[Marco Steinbach]

On Sat, 17 Nov 2018 08:42:18 +0800
Ben Woods <woodsb02 at gmail.com> wrote:

> On Sat, 17 Nov 2018 at 06:36, Marco Steinbach
> <coco at executive-computing.de> wrote:
> 
> > I'm trying to wrap my head around on how to avoid a zpool resilver
> > on a non-booting ZFS raidz1 of off four equally sized (GPT)
> > partitions on four distinct drives using eli for encyption.
> >
> > IOW: I do struggle with finding a way to attach all the
> > providers such, that ZFS does not initiate a resilver due to the
> > providers being attached sequentially.
> >  
> 
> 
> ZFS doesn't auto-mount by itself - there are other elements at work
> doing this.
> 1. During boot, the /etc/rc.d/zfs script will call "zfs mount -va" to
> automount any available datasets. Note this only happens once during
> boot (not continuously after boot), and it happens *after*
> the /etc/rc.d/geli script has already attached any geli providers.
> 2. The daemon zfsd(8) was newly introduced in FreeBSD 11.0 will
> online any vdevs as new GEOM devices appear. It also handles other
> devctl(4) events. This daemon is not enabled by default - you have to
> specify zfsd_enable="YES" in /etc/rc.conf if you want this behaviour.
>
> 
> I suspect your problem is the second items here. Have you enabled it?

I think what I was seeing was the result of me not being aware of the
geli rc script, and the rc order at boot with regards to zfs.

I didn't use zfsd -- I am not even sure I was aware of its existance.

From the man page of zfsd it looks like it'll help with what until
now was my main worry with ZFS on FreeBSD: Commands not returning,
forcing a reboot, due to devices dying or erroring out.  I need to
check that with pulling a (USB) drive from under zfs's feet :)

> I've created and initialized the partitions as follows (generic
> > notation, comments on chosen encryption algo welome, since this
> > testing setup lacks AES-NI):
> > # gpart create -s gpt /dev/ada[2-5]
> > # gpart add -t freebsd-zfs /dev/ada[2-5]
> > # geli init -e AES-CBC -l 128 /dev/ada[2-5]p1
> >  
> 
> I suspect you are just using the [2-5] as shorthand in your email
> here, and not running this exact command on FreeBSD 11.2.

Yes, that was shorthand notation.

> For reference, geli has been updated on FreeBSD 12.0 such that you
> could run this exact command to init multiple providers in a single
> command (they will all use the same password/file with unique salt).
> 
> 
> > Then I attached the geli partitions like so:
> > # geli attach /dev/ada[2-5]p1
> >  
> 
> Same for geli attach - this could all be done in the one command like
> this on FreeBSD 12.0.
> 
> 
> > And finally created a raidz1 spanning all four partitions:
> > # zpool create u0001 raidz1 /dev/ada[2-5]p1.eli
> >
> > That works flawlessly. And naturally, after a reboot none of the
> > encrypted devices is available to the zpool then, unless I attach
> > them. 
> 
> You can configure the /etc/rc.d/geli script to attach additional
> devices (e.g. non-root ZFS) during userland boot. You can add the
> following to /etc/rc.conf:
> geli_devices="ada2p1 ada3p1 ada4p1 ada5p1"
> This will prompt you to type your password in 4 times during userland
> boot, and each would be attached one at a time.
> 
> In FreeBSD 12.0, you can make this only prompt you for your password
> once by putting the following in /etc/rc.conf:
> geli_groups="u0001"
> geli_u0001_devices="ada2p1 ada3p1 ada4p1 ada5p1"
> 

Pure gold. I just tried this back and forth on 12-RC1 (and 11.2 minus
the neat provider name expansion), and it does exactly what I'm after.
 
> >
> > Doing so using geli attach requires me to attach them sequentially,
> > which then results in ZFS resilvering the pool.
> >  
> 
> This would only occur if there was something automatically onlining
> the vdevs and mounting the datasets, such as zfsd.
> 
> 
> > So, here's my questions:
> >
> > 1. Is the inavoidable resilver intended behaviour based on current
> > implementation, or am I missing something ?
> >  
> 
> I suspect this is zfsd working against you in this instance.
> 
> 2. In case I use a bootable zfsroot (cudos to allanjude@, I highly
> > recommend his BSDCan presentations on the matter), is there a way to
> > hand over the zfsroot passphrase to eli for automatically attaching
> > other providers ?
> >  
> 
> The geli attach for root ZFS devices would be handled by the boot
> loader, whilst the geli attach for non-root ZFS devices should be
> handled by the userland /etc/rc.d/geli script.
> Whilst you will need to type in your password separately for root and
> non-root devices, FreeBSD 12.0 at least makes it so you don't have to
> type the password multiple times for non-root devices.
> There might be a way to get the boot loader to handle the non-root
> devices also - I don't have experience with this.

I think I'll start by looking at how encrpyted swap is mounted --
since, oddly, I have 11.2 occassionally ask me again for the eli swap
providers passphrase during boot from a encrypted zroot. The system was
installed using the encrypted swap and zroot option of the installer. 

> > Please note, that I'd like to stick as close as possible to what the
> > base system offers for this use-case.
> >
> > MfG CoCo
> >  
> 
> 
> Good luck!

Thank you for your explanations -- much appreciated.  I don't think
that stack of new SMR drives likes the look on my face -- time for some
encrypted cold storage testing ...

MfG CoCo