ZFS - Unable to replace replacing drive

[Steven Schlansker]

It's a total pain that it doesn't work as you expect.
Last time this happened to me, I fixed it by dd'ing a zfs label from one of the failed drives (only needed to grab a few KB, which it managed to do even though it was broken) onto a new drive and then letting it finish the resilver process.
If your dead drives are still alive enough to grab the metadata from, that could solve your problem.

On Jul 11, 2010, at 3:38 PM, S M wrote:

> 
> Hi all,
> 
> Can anyone help with a ZFS problem I've been having? I have a single ZFS pool as shown below:
> 
> # zpool status
>  pool: vault
> state: DEGRADED
> scrub: resilver completed after 0h8m with 0 errors on Sun Jul 11 22:18:27 2010
> config:
> 
>        NAME                       STATE     READ WRITE CKSUM
>        vault                      DEGRADED     0     0     0
>          mirror                   ONLINE       0     0     0
>            da0.eli                ONLINE       0     0     0
>            da8.eli                ONLINE       0     0     0
>            ad4.eli                ONLINE       0     0     0
>          mirror                   DEGRADED     0     0     0
>            da1.eli                ONLINE       0     0     0  10.0M resilvered
>            replacing              UNAVAIL      0 7.42K     0  insufficient replicas
>              176228571956779269   UNAVAIL      0 7.79K     0  was /dev/ad6.eli
>              1650754158582652028  UNAVAIL      0 7.81K     0  was /dev/da99.eli
>            da9.eli                ONLINE       0     0     0  10.0M resilvered
> 
> errors: No known data errors
> 
> However I can't seem to get rid of the replacing drives. What happened was that I tried to replace /dev/ad6.eli with /dev/da99.eli (because ad6.eli broke) and da99.eli was also faulty and so failed the replace.
> 
> How do I get rid of them? The array is otherwise healthy....
> 
> # zpool offline vault 176228571956779269
> cannot offline 176228571956779269: no valid replicas
> # zpool offline vault 1650754158582652028
> cannot offline 1650754158582652028: no valid replicas
> # zpool detach vault 1650754158582652028
> cannot detach 1650754158582652028: no valid replicas
> # zpool detach vault 176228571956779269
> cannot detach 176228571956779269: no valid replicas
> 
> replace doeesn't work either.
> 
> ZDB returns:
> 
> vault
>    version=14
>    name='vault'
>    state=0
>    txg=173963
>    pool_guid=14228437791114878229
>    hostid=4269824854
>    hostname='vault.internal'
>    vdev_tree
>        type='root'
>        id=0
>        guid=14228437791114878229
>        children[0]
>                type='mirror'
>                id=0
>                guid=15203813302661836458
>                metaslab_array=23
>                metaslab_shift=34
>                ashift=12
>                asize=2000394125312
>                is_log=0
>                children[0]
>                        type='disk'
>                        id=0
>                        guid=7238937831460639650
>                        path='/dev/da0.eli'
>                        whole_disk=0
>                        DTL=51
>                children[1]
>                        type='disk'
>                        id=1
>                        guid=14125934351009663627
>                        path='/dev/da8.eli'
>                        whole_disk=0
>                        DTL=53
>                children[2]
>                        type='disk'
>                        id=2
>                        guid=8553277371515810839
>                        path='/dev/ad4.eli'
>                        whole_disk=0
>                        DTL=169
>        children[1]
>                type='mirror'
>                id=1
>                guid=14672070090293526603
>                metaslab_array=118
>                metaslab_shift=34
>                ashift=12
>                asize=2000394125312
>                is_log=0
>                children[0]
>                        type='disk'
>                        id=0
>                        guid=4020825290461516758
>                        path='/dev/da1.eli'
>                        whole_disk=0
>                        DTL=123
>                children[1]
>                        type='replacing'
>                        id=1
>                        guid=496235429895718168
>                        whole_disk=0
>                        children[0]
>                                type='disk'
>                                id=0
>                                guid=176228571956779269
>                                path='/dev/ad6.eli'
>                                whole_disk=0
>                                not_present=1
>                                DTL=54
>                        children[1]
>                                type='disk'
>                                id=1
>                                guid=1650754158582652028
>                                path='/dev/da99.eli'
>                                whole_disk=0
>                                not_present=1
>                                DTL=168
>                children[2]
>                        type='disk'
>                        id=2
>                        guid=5694121814811722857
>                        path='/dev/da9.eli'
>                        whole_disk=0
>                        DTL=120
> 
> 
> uname -a: FreeBSD vault.internal 8.1-PRERELEASE FreeBSD 8.1-PRERELEASE #4: Thu Jul  1 09:47:23 UTC 2010     sm at vault.internal:/usr/obj/usr/src/sys/vault  amd64
> 
> How do I fix this? ZFS says that the array is degraded but it isn't really, I just can't get rid of this.
> 		 	   		  
> _________________________________________________________________
> http://clk.atdmt.com/UKM/go/197222280/direct/01/
> We want to hear all your funny, exciting and crazy Hotmail stories. Tell us now_______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"