Any plans for EC2 AMIs with RootOnZFS?

[Colin Percival]

Hi Ben,

On 1/5/19 4:39 AM, Ben Woods wrote:
> On Fri, 28 Dec 2018 at 12:43, Colin Percival <cperciva at tarsnap.com> wrote:
>> I've created an experimental FreeBSD 12.0 ZFS AMI in the us-east-1 region:
>> ami-0786f5b55d5aa573f
> 
> Thanks a lot for doing this - much appreciated.
> 
> I have just given it a try, and have the following feedback for you:
> 
> 1. ZFS DATASETS AND PROPERTIES SHOULD MATCH INSTALLER DEFAULT

Oops.  I thought I was doing this, but I guess the wiki page I was following
(https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot) didn't quite match...

> Looking at how you created the zpool and zfs datasets (using the "zpool
> history zroot" command), the following changes would achieve this outcome:
> [snip]

Thanks!

> 2. LOADER.CONF SETTINGS
> The following settings should be included in /boot/loader.conf:
> kern.geom.label.disk_ident.enable="0"
> kern.geom.label.gptid.enable="0"
> 
> 3. SYSCTL.CONF SETTINGS
> The following settings should be included in /etc/sysctl.conf:
> vfs.zfs.min_auto_ashift=12

Got it.

> 4. THE GPT TABLE IS INITIALLY CORRUPT
> I suspect this is because I used a 50Gb disk, and it hasn't been expanded
> correctly.
> The following error was in DMESG and I suspect is related:
> GEOM: ada0: the secondary GPT header is not in the last LBA.

Yes.  This message appears whenever you launch an instance with a
larger-than-default root disk; with UFS, the AMI promptly fixes the GPT and
resizes the
UFS filesystem.

We can't avoid having the message (well, not without fiddling with the kernel
bits to silence the warning) but we should fix the autoresizing to work with
ZFS as well as UFS.

> I was able to fix this with the following commands on firstboot:
> # gpart recover ada0
> # gpart resize -i 2 ada0
> # zpool online -e zroot ada0p2

Can you look at /etc/rc.d/growfs (which currently only works for UFS, using
the 'growfs' utility) and see if you can teach it to handle ZFS filesystems?

Alternatively, maybe someone can teach growfs(8) to say "hey, this looks like
a disk with ZFS... let's run 'zpool online -e' instead"?

Either way, I don't think this is something I can fix for a FreeBSD 12.0 ZFS
AMI, but if the growfs rc.d script can be made to do the right thing with ZFS
then at least future releases will work.

-- 
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid