zpool import -- state: UNAVAIL, status: One or more devices are missing

[David Christensen]

On 5/20/19 7:17 PM, David Christensen wrote:

> I exported the pool per "Migrating ZFS Storage Pools":
> 
> https://docs.oracle.com/cd/E19253-01/819-5461/gbchy/index.html
> 
> root at dipsy:~ # zpool export tracy_p0

> But when I try to import the pool:
> 
> 2019-05-20 17:21:21 toor at dipsy ~
> # zpool import
>     pool: tracy_p0
>       id: 6957586590476290317
>    state: UNAVAIL
>   status: One or more devices are missing from the system.
>   action: The pool cannot be imported. Attach the missing
>      devices and try again.
>     see: http://illumos.org/msg/ZFS-8000-3C
>   config:
> 
>      tracy_p0                  UNAVAIL  insufficient replicas
>        raidz1-0                UNAVAIL  insufficient replicas
>          8533131415900712652   UNAVAIL  cannot open
>          11863649002236717295  UNAVAIL  cannot open
>          16410494706036382010  UNAVAIL  cannot open

I found the solution:

     Reboot SSD.  When the following command is issued (via SSH):

	2019-05-20 23:46:22 toor at dipsy ~
	# zpool import
	   pool: tracy_p0
	     id: 6957586590476290317
	  state: UNAVAIL
	 status: One or more devices are missing from the system.
	 action: The pool cannot be imported. Attach the missing
		devices and try again.
	   see: http://illumos.org/msg/ZFS-8000-3C
	 config:

		tracy_p0                  UNAVAIL  insufficient replicas
		  raidz1-0                UNAVAIL  insufficient replicas
		    8533131415900712652   UNAVAIL  cannot open
		    11863649002236717295  UNAVAIL  cannot open
		    16410494706036382010  UNAVAIL  cannot open

     The following appears on the console:

	GEOM_ELI: Device ada2s1.eli destroyed.
	GEOM_ELI: Detached ada2s1.eli on last close.
	GEOM_ELI: Device ada1s1.eli destroyed.
	GEOM_ELI: Detached ada211.eli on last close.
	GEOM_ELI: Device ada3s1.eli destroyed.
	GEOM_ELI: Detached ada3s1.eli on last close.

     Suspect that some boot loader gives ZFS one chance to connect to
     GELI drives and, if ZFS fails, GELI will nuke the *.eli providers
     whenever the next thing tries to access them ('zpool import') (?).

     Look at /etc/rc.conf again for clue as to how to attach GELI
     providers:

	2019-05-20 23:46:27 toor at dipsy ~
	# grep ada /etc/rc.conf
	geli_ada1s1_flags="-k /root/raidz-geli.key -p"
	geli_ada2s1_flags="-k /root/raidz-geli.key -p"
	geli_ada3s1_flags="-k /root/raidz-geli.key -p"
	geli_devices="ada1s1 ada2s1 ada3s1"

     Attach them manually:

	2019-05-21 00:00:11 toor at dipsy ~
	# geli attach -k /root/raidz-geli.key -p -v ada1s1
	Attached to ada1s1.
	Done.

	2019-05-21 00:01:25 toor at dipsy ~
	# geli attach -k /root/raidz-geli.key -p -v ada2s1
	Attached to ada2s1.
	Done.

	2019-05-21 00:01:27 toor at dipsy ~
	# geli attach -k /root/raidz-geli.key -p -v ada3s1
	Attached to ada3s1.
	Done.

	2019-05-21 00:02:28 toor at dipsy ~
	# geli status
	       Name  Status  Components
	ada0s1d.eli  ACTIVE  ada0s1d
	 ada1s1.eli  ACTIVE  ada1s1
	 ada2s1.eli  ACTIVE  ada2s1
	 ada3s1.eli  ACTIVE  ada3s1

	2019-05-21 00:02:31 toor at dipsy ~
	# ls /dev/ada*
	/dev/ada0		/dev/ada0s1d.eli	/dev/ada2s1
	/dev/ada0s1		/dev/ada1		/dev/ada2s1.eli
	/dev/ada0s1a		/dev/ada1s1		/dev/ada3
	/dev/ada0s1b		/dev/ada1s1.eli		/dev/ada3s1
	/dev/ada0s1d		/dev/ada2		/dev/ada3s1.eli

     Import zpool:

	2019-05-21 00:03:43 toor at dipsy ~
	# zpool import
	   pool: tracy_p0
	     id: 6957586590476290317
	  state: ONLINE
	 action: The pool can be imported using its name or numeric identifier.
	 config:

		tracy_p0        ONLINE
		  raidz1-0      ONLINE
		    ada3s1.eli  ONLINE
		    ada1s1.eli  ONLINE
		    ada2s1.eli  ONLINE

	2019-05-21 00:04:30 toor at dipsy ~
	# zpool import tracy_p0

	2019-05-21 00:04:51 toor at dipsy ~
	# zpool list
	NAME            SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP 
DEDUP  HEALTH  ALTROOT
	<snip>
	tracy_p0       4.06T  1.34T  2.72T        -         -     0%    33% 
1.00x  ONLINE  -


Easy once you know how!  ;-)


David