zpool import -- state: UNAVAIL, status: One or more devices are missing

[David Christensen]

freebsd-questions:

I have a computer with FreeBSD:

2019-05-20 16:59:02 toor at dipsy ~
# freebsd-version ; uname -a
11.2-RELEASE-p9
FreeBSD dipsy 11.2-RELEASE-p9 FreeBSD 11.2-RELEASE-p9 #0: Tue Feb  5 
15:30:36 UTC 2019 
root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64


The computer had a 16 GB USB 3.0 flash drive as the system drive, with 
ZFS boot, GELI swap, and GELI ZFS root, all created by the FreeBSD 11.2 
RELEASE amd64 installer "Auto (ZFS)".


The computer has three 1.5 TB HDD's with GELI ZFS RAIDZ1:

root at dipsy:~ # zpool list
NAME            SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP 
DEDUP  HEALTH  ALTROOT
<snip>
tracy_p0       4.06T  1.34T  2.72T        -         -     0%    33% 
1.00x  ONLINE  -


I exported the pool per "Migrating ZFS Storage Pools":

https://docs.oracle.com/cd/E19253-01/819-5461/gbchy/index.html

root at dipsy:~ # zpool export tracy_p0


Then I copied the raw USB system drive image to an SSD.


The machine would not boot -- it appeared that the boot loader could not 
find the root partition.


I booted the FreeBSD 11.2 installer to a rescue shell, mounted the SSD 
boot partition, and looked around.  I found that /boot/loader.conf had 
some variables containing the USB device node 'da0', so I changed them 
to the SSD device node 'ada0':

2019-05-20 17:00:19 toor at dipsy ~
# cat /boot/loader.conf | grep ada
geli_ada0s1d_keyfile0_load="YES"
geli_ada0s1d_keyfile0_type="ada0s1d:geli_keyfile0"
geli_ada0s1d_keyfile0_name="/boot/encryption.key"


Somehow, I discovered that I also needed to move aside 
/boot/ssd/zpool.cache.


Now the machine can boot from the SSD, but the RAIDZ1 was missing.
Looking in /etc/rc.conf, I saw some variables containing the HDD device 
nodes 'ada0', 'ada1', and 'ada2', so I changed them to the new device 
nodes 'ada1, 'ada2', and 'ada3':

2019-05-20 17:22:00 toor at dipsy ~
# cat /etc/rc.conf | grep ada
geli_ada1s1_flags="-k /root/raidz-geli.key -p"
geli_ada2s1_flags="-k /root/raidz-geli.key -p"
geli_ada3s1_flags="-k /root/raidz-geli.key -p"
geli_devices="ada1s1 ada2s1 ada3s1"


After rebooting again, FreeBSD could now see the all the drives, 
partitions, and GELI encrypted disk devices:

2019-05-20 17:15:01 toor at dipsy ~
# camcontrol devlist | grep ada
<SAMSUNG SSD UM410 Series 2.5" 16GB VAM12D1Q>  at scbus0 target 0 lun 0 
(pass0,ada0)
<ST31500341AS CC1H>                at scbus2 target 0 lun 0 (pass1,ada1)
<ST31500341AS CC1H>                at scbus3 target 0 lun 0 (pass2,ada2)
<ST31500341AS CC1H>                at scbus4 target 0 lun 0 (pass3,ada3)

2019-05-20 17:15:53 toor at dipsy ~
# geli status
        Name  Status  Components
ada0s1d.eli  ACTIVE  ada0s1d
  ada1s1.eli  ACTIVE  ada1s1
  ada2s1.eli  ACTIVE  ada2s1
  ada3s1.eli  ACTIVE  ada3s1

2019-05-20 17:16:08 toor at dipsy ~
# ls -1 /dev/ada*
/dev/ada0
/dev/ada0s1
/dev/ada0s1a
/dev/ada0s1b
/dev/ada0s1d
/dev/ada0s1d.eli
/dev/ada1
/dev/ada1s1
/dev/ada1s1.eli
/dev/ada2
/dev/ada2s1
/dev/ada2s1.eli
/dev/ada3
/dev/ada3s1
/dev/ada3s1.eli


But when I try to import the pool:

2019-05-20 17:21:21 toor at dipsy ~
# zpool import
    pool: tracy_p0
      id: 6957586590476290317
   state: UNAVAIL
  status: One or more devices are missing from the system.
  action: The pool cannot be imported. Attach the missing
	devices and try again.
    see: http://illumos.org/msg/ZFS-8000-3C
  config:

	tracy_p0                  UNAVAIL  insufficient replicas
	  raidz1-0                UNAVAIL  insufficient replicas
	    8533131415900712652   UNAVAIL  cannot open
	    11863649002236717295  UNAVAIL  cannot open
	    16410494706036382010  UNAVAIL  cannot open


Following the 'see' link, STFW, RTFM, etc., I am unable to determine how 
to attach the drives.


Furthermore, if I boot the original USB system drive and try to import 
the RAIDZ1:

2019-05-20 18:48:43 toor at dipsy ~
# zpool import
    pool: tracy_p0
      id: 6957586590476290317
   state: UNAVAIL
  status: One or more devices are missing from the system.
  action: The pool cannot be imported. Attach the missing
	devices and try again.
    see: http://illumos.org/msg/ZFS-8000-3C
  config:

	tracy_p0                  UNAVAIL  insufficient replicas
	  raidz1-0                UNAVAIL  insufficient replicas
	    8533131415900712652   UNAVAIL  cannot open
	    11863649002236717295  UNAVAIL  cannot open
	    16410494706036382010  UNAVAIL  cannot open


Any suggestions?


David