Suggestions for working with unstable nvme dev names in AWS

[Matthias Oestreicher]

Am Dienstag, den 14.05.2019, 12:24 -0700 schrieb George Hartzell:
> Polytropon writes:
>  > On Tue, 14 May 2019 08:59:01 -0700, George Hartzell wrote:
>  > > Matthew Seaman writes:
>  > >  > [...] but if you
>  > >  > are using ZFS, then shuffling the disks around should not make any
>  > >  > difference. 
>  > >  > [...]
>  > > Yes, once I have them set up (ZFS or labeled), it doesn't matter what
>  > > device names they end up having.  For now I just do the setup by hand,
>  > > poking around a bit.  Same trick in the Linux world, you end up
>  > > referring to them by their UUID or ....
>  > 
>  > In addition to what Matthew suggested, you could use UFS-IDs
>  > in case the disks are initialized with UFS. You can find more
>  > information here (at the bottom of the page):
>  > [...]
> 
> Yes.  As I mentioned in my response to Matthew, once I have some sort
> of filesystem/zpool on the device, it's straightforward (TMTOWTDI).
> 
> The problem is being able to provision the system automatically
> without user intervention.
> 
> In the Linux world, I can use e.g. Terraform to set up a pair of
> additional volumes and tell it to call them `/dev/sdy` and `/dev/sdz`.
> The Linux magic happens and I get pair of symlinks that I can use in
> my e.g. Ansible playbooks, that point to whatever the devices came up
> as when it booted.  I build filesystems on the devices, add them via
> their UUID's to `/etc/fstab` and I'm off and running.
> 
> I can't [seem to] do this in the FreeBSD world; even if I name the
> devices `/dev/nvme1` (the fast and big one) and `/dev/nvme2` (the slow
> and small one), there's no guarantee that they'll have those names
> when the machine boots.
> 
> This is a weirdly AWS issue and their peace offering is to stash the
> requested device name in the device/controller/"hardware" and provide
> a tool that digs it out.
> 
> I'm trying to figure out what I can do about it from FreeBSD.  Perhaps
> there's already a solution.  Perhaps the nvme driver needs to be
> extended to provide access to the magic AWS info stash and then
> something like Amazon Linux's `ebsnvme-id` can pry it out.
> 
> g.
Hei,
I'm not familiar with Amazon's AWS, but if your only problem is shiftig device
names for UFS filesystems, then on modern systems, GPT labels is the way to go.
There has been a lot of confusion over the years, about the many ways to apply
different types of labels to devices on FreeBSD, but really GEOM labels, UUIDs,
etc, are only useful on old systems where there's no support for GPT.

GPT labels are only applied to partitions, not whole drives, but they are extremely
flexible. They can be applied and changed at any time, even on mounted filesystems.
In comparison to GEOM labels and all other ID types, they will never be hidden if
the devices original device name (like nvm0 or nvm1) is in use.
At any time will 'gpart show -l' show the GPT labels you applied, and they can be
used to manually mount and in /etc/fstab.
I have never used any other labels for years and even disables all others in

/boot/loader.conf
kern.geom.label.disk_ident.enable=0
kern.geom.label.gptid.enable=0
kern.geom.label.ufsid.enable=0

You can apply a GPT label with
# gpart modify -l mylabel -i N /dev/nvm1

and then add something like the following to /etc/fstab
/dev/gpt/mylabel       /       ufs     rw      1       1

There is only a single limitation with GPT labels and that is they don't work
when you use UFS journaling via GEOM, as the GPT label will be the same for e.g
/dev/nvm0p1 and /dev/nvm0p1.journal.

Another big plus is, they work with every partition type, freebsd-ufs, freebsd-boot,
swap, EFI, freebsd-zfs...
One label type for everything can avoid some headache imo.

Hope that clears up some confusion.
Matthias



> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org"