Device names in ZFS pool

[Scott Bennett]

     On Thu, 9 Sep 2021 16:48:28 -0400 Chris Ross <cross+freebsd at distal.com>
wrote:

>Hey there.  So, I created a ZFS pool, a RAID-Z with ?da5 da2 da3?.  These are then the device names that appeared in ?zpool status mypool?.  However, da3 had errors, and I replaced it by another disk living at da1.  Then, I removed the bad disk and moved things in the enclosure.  But, after this, some ids seem to have changed, and what was originally ?da5? is now ?da4? and ?zpool status mypool? shows:
>
     FreeBSD normally assigns da device numbers in the order that the devices
are detected and identified by the kernel.  ada device enumeration appears to
happen in the order encountered by the bus-controller-port hardware addresses
of the devices.  da devices are today most commonly devices attached by USB.
     Because you gave entire drives to ZFS, your label options are limited to
/dev/da?, /dev/diskid/*, and maybe /dev/gptid/*.  For the latter, add the
following line to /boot/loader.conf.  If you had used gpart(8) to partition
the devices with GPT layouts, then you would have had the option of using gpt
labels as well, which would have appeared in /dev/gpt.  For my largest pool,
for which I use the first partition on each of six drives (it's a raidz2), I
use the partition label to identify the device by its manufacturer and serial
number, which makes identifying a problem device both trivial and exact.  I
don't even need to apply an exterior printed label because the manufacturer
has already done so.

kern.geom.label.gptid.enable="1"

Because the first of those is subject to frequent change upon reboot or
detaching and reattaching the device in question, one of the latter is
preferable in many situations and especially those involving ZFS.  However,
it is worth noting that a diskid will change anytime you replace a device
in ZFS.  In any case, whenever is device in a ZFS pool is offline or faulted,
a zpool status command will display that device as a many-digit integer and
as, for example, "(was /dev/da5)".  In that situation, trying to do, say, a

zpool replace mypool da5

or a

zpool replace mypool da5 da8

may fail because da5 is not available.  It will fail when the device that was
on /dev/da5 has been enumerated as, say, da6 on the latest reboot.  Similarly,
a

zpool replace mypool da8

will fail because da8 is/was not part of the pool and is not the device that
ZFS is looking for.  However, if the number shown in the status output is
112574468553112, then a

zpool replace mypool 112574468553112 da8

is what you need.
     I mostly let ZFS do whatever it wants w.r.t. device names and then just
do my best to accommodate it because fighting it rarely succeeds. :-}  That
said, I would note that many folks prefer to refer to a device when creating
a pool or adding/attaching/removing a device by its gptid.  I haven't yet done
this, so I can't say for sure that this will work for you, but it may cause
ZFS to prefer the gptid when recognizing devices while activating or importing
a pool, in which case you will have a consistent way to identify each devices
in the pool.  Many have said, OTOH, that reference by diskid allows them to
identify a failed or failing physical device more easily because they can
paste a matching label onto the drive.

>	NAME                                  STATE     READ WRITE CKSUM
>	mypool                                ONLINE       0     0     0
>	  raidz1-0                            ONLINE       0     0     0
>	    diskid/DISK-Z1Z946N90000R546NBZH  ONLINE       0     0     0

     You can, of course, refer to /dev/diskid devices if you like.  I don't
like having them around, mainly because ZFS seems to refer to devices with
multiple names without any consistency of order.  If you don't like them
being around, you can disable their generation by placing the following line
into /boot/loader.conf.

kern.geom.label.disk_ident.enable="0"

My largest pool only has three drives that are connected (usually) by USB
externally, and its other three are on ada devices, whose addresses do not
change, so I haven't really had many problems in identifying the drive(s) of
interest.  I also disable the generation of gptid device names for the same
reasons.

>	    da2                               ONLINE       0     0     0
>	    da1                               ONLINE       0     0     0
>
>If I use zdb, I can see that value stored in the ?path? of each child of the vdev/pool.  But, can I _change_ it?  I know this is the reason many people suggest not using device names like I did, but, if I switch to gpt labels or the like, I?ll still need to update these in this existing vdev, right?

     Please be aware that you can only switch to gpt labels by partitioning
the drives as GPT-partitioned drives with at least one partition on each
that you have labeled.  (See gpart(8)'s add and modify subcommands.)  And
you will only be able to do that in your case by destroying the pool first,
partitioning the drive, and then recreating the pool because you have
already given the *entire* drive to ZFS.  You have left no space for the
partition tables, and you cannot reduce the size of a device node already
given to ZFS.
>
>How can I change these path labels?

     1.  You can reboot, and then see how the device names are assigned
         that time.

     2.  You can export the pool and then import it with, for example,
         zpool import -d /dev/diskid mypool
         or
         zpool import -d /dev/gptid mypool
         to get ZFS to look for ZFS labels on those devices instead of
         starting its search in /dev.

However, if your pool is small, which it presumably is because it is a
raidz1 pool, why worry about which device names ZFS uses?  You should have
little trouble figuring out what it is doing or how to deal with a problem.
OTOH, when your pool comprises a dozen or more devices, then it may well be
worth using diskid or gptid device names and placing printed or handwritten
labels onto the drives.


                                  Scott Bennett, Comm. ASMELG, CFIAG
**********************************************************************
* Internet:   bennett at sdf.org   *xor*   bennett at freeshell.org  *
*--------------------------------------------------------------------*
* "A well regulated and disciplined militia, is at all times a good  *
* objection to the introduction of that bane of all free governments *
* -- a standing army."                                               *
*    -- Gov. John Hancock, New York Journal, 28 January 1790         *
**********************************************************************