Suggestions for working with unstable nvme dev names in AWS

[Matthias Oestreicher]

Am Dienstag, den 14.05.2019, 14:01 -0700 schrieb George Hartzell:
> Matthias Oestreicher writes:
>  > Am Dienstag, den 14.05.2019, 12:24 -0700 schrieb George Hartzell:
>  > > Polytropon writes:
>  > >  > On Tue, 14 May 2019 08:59:01 -0700, George Hartzell wrote:
>  > >  > > Matthew Seaman writes:
>  > >  > >  > [...] but if you
>  > >  > >  > are using ZFS, then shuffling the disks around should not make any
>  > >  > >  > difference. 
>  > >  > >  > [...]
>  > >  > > Yes, once I have them set up (ZFS or labeled), it doesn't matter what
>  > >  > > device names they end up having.  For now I just do the setup by hand,
>  > >  > > poking around a bit.  Same trick in the Linux world, you end up
>  > >  > > referring to them by their UUID or ....
>  > >  > 
>  > >  > In addition to what Matthew suggested, you could use UFS-IDs
>  > >  > in case the disks are initialized with UFS. You can find more
>  > >  > information here (at the bottom of the page):
>  > >  > [...]
>  > > 
>  > > Yes.  As I mentioned in my response to Matthew, once I have some sort
>  > > of filesystem/zpool on the device, it's straightforward (TMTOWTDI).
>  > > 
>  > > The problem is being able to provision the system automatically
>  > > without user intervention.
>  > > [...]
>  > Hei,
>  > I'm not familiar with Amazon's AWS, but if your only problem is shiftig device
>  > names for UFS filesystems, then on modern systems, GPT labels is the way to go.
>  > [...]
> 
> Yes, yes, and yes.  I do appreciate all of the answers but I
> apparently haven't made clear the point of my question.  I think that
> you've all explained ways that I can log in and set things up manually
> so that things work as they should for the rest of time.
> 
> You (Matthias) suggested that I could just:
> 
> > ```
> > # gpart modify -l mylabel -i N /dev/nvm1
> > ```
> 
> But how do I know which of the devices is the one that I'd like
> labeled 'mylabel' and which is the one that I'd like labeled 'blort'?
> 
> Another way to explain my situation might be to ask how I can automate
> applying the labels.
> 
> Imagine that in my automated creation of the instance, I asked for two
> additional devices, a big-slow one which I asked to be called
> `/dev/sdh` and a small-fast one that I asked to be called `/dev/sdz`.
> 
> But when I boot, I find that I have two devices (in addition to the
> root device), `/dev/nvme1` and `/dev/nvme2`.  There's no way to know
> which is the big-slow one that I wanted to call `/dev/sdh` and which
> is the small-fast `/dev/sdz`.  In fact, if I reboot the machine,
> sometimes the big-slow one will be `/dev/nvme1` and sometimes it will
> be `/dev/nvme2`.
> 
> Given that situation, how do you write an automated script that will
> label the big-slow one `backups` and the small-fast one `speedy`?
> 
> In the Linux world, `ebsnvme-id` & `udev` rules create symlinks at
> boot time that link the names that I requested to whatever the device
> is currently named.  That makes writing the script easy.
> 
> We lack `ebsnvme-id` and our nvme driver doesn't seem to have any
> knowledge of AWS' tricksy trick.  Or perhaps not and I've just missed
> out how we do it.
> 
> Thanks (seriously) for all the answer,
> 
> g.
I have to admit that I'm still a bit unsure if I understand your problem.

You are worried about, that the big-slow and the small-fast change their
device names when the system boots...
The GPT labels I suggested will survive a reboot, so no need to run a script
each time the system boots, to reapply those labels to the right drive.

What you only need to do once, is to determine which /dev/nvmN is the big-slow
one and which the small-fast. Then you apply your labels, e.g.:

	# gpart modify -l big-slow -i 2 /dev/nvm1 
	# gpart modify -l small-fast -i 2 /dev/nvm0

(you could post the output from 'gpart show' so I can give a more precise example)
Then you need to edit your /etc/fstab accordingly, e.g. (what I assume)

/dev/gpt/small-fast       /       ufs     rw      1       1
/dev/gpt/big-slow         /data   ufs     rw      1       1

Then there's no need to create symlinks to the right device either.
FreeBSD actually has devfs(8) to do that, but you don't need it here.

If you for some reason need to write a script that automates such things, below are some examples for different use cases.

------------------------------------------------------------------------
The best way for me to get the device name of a specific drive is:

	# camcontrol devlist | grep "^<Name of small and fast>" | cut -d "(" -f2 | cut -d "," -f1
------------------------------------------------------------------------
Or to do something with all attached nvmN drives

	drives=$( sysctl -n kern.disks | tr ' ' '\n' | grep -e '^nvm[[:digit:]]' | sort )

...and then process $drives in a loop.
------------------------------------------------------------------------
Last but not least, an easy way to process associations between drive names and existing labels:

	associations=$( geom label status | awk '/ nvm[[:digit:]]/{ print $3, substr($1,5,20) }' | tr ' ' ':' )
	
This gives you a list of all <label>:<device name> in $associations that you then can easily process in a loop.
------------------------------------------------------------------------

But as said, if there isn't any Amazon AWS weirdness involved, simply applying GPT labels and editing /etc/fstab should be enough to stop drive names from moving around.