ZFS weird issue...

Michelle Sullivan michelle at sorbs.net
Sat Dec 6 02:21:34 UTC 2014


Steven Hartland wrote:
> Formatting for you zpool list looks odd, making it not clear if its
> your spare which is unavailable or one of your pool drives.
>
> If its the spare, try zpool remove.

root at colossus:~ # zpool remove sorbs mfid14
cannot remove mfid14: only inactive hot spares, cache, top-level, or log
devices can be removed


The 'UNAVAIL' is the new drive.

Michelle
>
> On 06/12/2014 01:40, Michelle Sullivan wrote:
>> Here's what happened:
>>
>> LSI9260-16i, 16x3T SATA drives.  15 LDs each configured as single disk
>> RAID0 last drive a hot spare.
>>
>> All LDs configured as a ZFS RaidZ2 pool ('sorbs')
>>
>> Bay/Drive 8 failed, hot spare kicked in.  ZFS resilvered. (mfid15 became
>> spare-8)
>>
>> On reboot mfid9-15 were re-named automatically to mfid8-14 ... ZFS
>> didn't seems to care...
>>
>> Days later new drive to replace the dead drive arrived and was
>> inserted.  System refused to re-add as there was data in the cache, so
>> rebooted and cleared the cache (as per many on web faq's)  Reconfigured
>> it to match the others.  Can't do a zpool replace mfid8 because that's
>> already in the pool... (was mfid9) can't use mfid15 because zpool
>> reports it's not part of the config... can't use the uniq-id it received
>> (can't find vdev) ... HELP!! :)
>>
>> This is what I can see currently ...
>>
>> Thanks, in advance.
>>
>> root at colossus:~ # zpool status -v
>>    pool: VirtualDisks
>>   state: ONLINE
>>    scan: none requested
>> config:
>>
>>      NAME                       STATE     READ WRITE CKSUM
>>      VirtualDisks               ONLINE       0     0     0
>>        zvol/sorbs/VirtualDisks  ONLINE       0     0     0
>>
>> errors: No known data errors
>>
>>    pool: sorbs
>>   state: DEGRADED
>> status: One or more devices could not be opened.  Sufficient replicas
>> exist for
>>      the pool to continue functioning in a degraded state.
>> action: Attach the missing device and online it using 'zpool online'.
>>     see: http://illumos.org/msg/ZFS-8000-2Q
>>    scan: scrub in progress since Fri Dec  5 17:11:29 2014
>>          2.51T scanned out of 29.9T at 89.4M/s, 89h7m to go
>>          0 repaired, 8.40% done
>> config:
>>
>>      NAME              STATE     READ WRITE CKSUM
>>      sorbs             DEGRADED     0     0     0
>>        raidz2-0        DEGRADED     0     0     0
>>          mfid0         ONLINE       0     0     0
>>          mfid1         ONLINE       0     0     0
>>          mfid2         ONLINE       0     0     0
>>          mfid3         ONLINE       0     0     0
>>          mfid4         ONLINE       0     0     0
>>          mfid5         ONLINE       0     0     0
>>          mfid6         ONLINE       0     0     0
>>          mfid7         ONLINE       0     0     0
>>          spare-8       DEGRADED     0     0     0
>>            1702922605  UNAVAIL      0     0     0  was /dev/mfid8
>>            mfid14      ONLINE       0     0     0
>>          mfid8         ONLINE       0     0     0
>>          mfid9         ONLINE       0     0     0
>>          mfid10        ONLINE       0     0     0
>>          mfid11        ONLINE       0     0     0
>>          mfid12        ONLINE       0     0     0
>>          mfid13        ONLINE       0     0     0
>>      spares
>>        933862663       INUSE     was /dev/mfid14
>>
>> errors: No known data errors
>> root at colossus:~ # uname -a
>> FreeBSD colossus.sorbs.net 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898:
>> Thu Sep 26 22:50:31 UTC 2013
>> root at bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
>> root at colossus:~ # sh ./lsi.sh drives
>> Slot Number: 0 - Online, Spun Up
>> Slot Number: 1 - Online, Spun Up
>> Slot Number: 2 - Online, Spun Up
>> Slot Number: 3 - Online, Spun Up
>> Slot Number: 4 - Online, Spun Up
>> Slot Number: 5 - Online, Spun Up
>> Slot Number: 6 - Online, Spun Up
>> Slot Number: 7 - Online, Spun Up
>> Slot Number: 8 - Online, Spun Up
>> Slot Number: 9 - Online, Spun Up
>> Slot Number: 10 - Online, Spun Up
>> Slot Number: 11 - Online, Spun Up
>> Slot Number: 12 - Online, Spun Up
>> Slot Number: 13 - Online, Spun Up
>> Slot Number: 14 - Online, Spun Up
>> Slot Number: 15 - Online, Spun Up
>> root at colossus:~ # sh ./lsi.sh status
>>                                     
>> Adapter 0 -- Virtual Drive Information:
>> Virtual Drive: 0 (Target Id: 0)
>> Name                :
>> RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size                : 2.728 TB
>> Sector Size         : 512
>> Is VD emulated      : Yes
>> Parity Size         : 0
>> State               : Optimal
>> Strip Size          : 64 KB
>> Number Of Drives    : 1
>> Span Depth          : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy   : Enabled
>> Encryption Type     : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 1 (Target Id: 1)
>> Name                :
>> RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size                : 2.728 TB
>> Sector Size         : 512
>> Is VD emulated      : Yes
>> Parity Size         : 0
>> State               : Optimal
>> Strip Size          : 64 KB
>> Number Of Drives    : 1
>> Span Depth          : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy   : Enabled
>> Encryption Type     : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 2 (Target Id: 2)
>> Name                :
>> RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size                : 2.728 TB
>> Sector Size         : 512
>> Is VD emulated      : Yes
>> Parity Size         : 0
>> State               : Optimal
>> Strip Size          : 64 KB
>> Number Of Drives    : 1
>> Span Depth          : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy   : Enabled
>> Encryption Type     : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 3 (Target Id: 3)
>> Name                :
>> RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size                : 2.728 TB
>> Sector Size         : 512
>> Is VD emulated      : Yes
>> Parity Size         : 0
>> State               : Optimal
>> Strip Size          : 64 KB
>> Number Of Drives    : 1
>> Span Depth          : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy   : Enabled
>> Encryption Type     : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 4 (Target Id: 4)
>> Name                :
>> RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size                : 2.728 TB
>> Sector Size         : 512
>> Is VD emulated      : Yes
>> Parity Size         : 0
>> State               : Optimal
>> Strip Size          : 64 KB
>> Number Of Drives    : 1
>> Span Depth          : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy   : Enabled
>> Encryption Type     : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 5 (Target Id: 5)
>> Name                :
>> RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size                : 2.728 TB
>> Sector Size         : 512
>> Is VD emulated      : Yes
>> Parity Size         : 0
>> State               : Optimal
>> Strip Size          : 64 KB
>> Number Of Drives    : 1
>> Span Depth          : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy   : Enabled
>> Encryption Type     : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 6 (Target Id: 6)
>> Name                :
>> RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size                : 2.728 TB
>> Sector Size         : 512
>> Is VD emulated      : Yes
>> Parity Size         : 0
>> State               : Optimal
>> Strip Size          : 64 KB
>> Number Of Drives    : 1
>> Span Depth          : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy   : Enabled
>> Encryption Type     : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 7 (Target Id: 7)
>> Name                :
>> RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size                : 2.728 TB
>> Sector Size         : 512
>> Is VD emulated      : Yes
>> Parity Size         : 0
>> State               : Optimal
>> Strip Size          : 64 KB
>> Number Of Drives    : 1
>> Span Depth          : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy   : Enabled
>> Encryption Type     : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 8 (Target Id: 8)
>> Name                :
>> RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size                : 2.728 TB
>> Sector Size         : 512
>> Is VD emulated      : Yes
>> Parity Size         : 0
>> State               : Optimal
>> Strip Size          : 512 KB
>> Number Of Drives    : 1
>> Span Depth          : 1
>> Default Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy   : Disk's Default
>> Encryption Type     : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 9 (Target Id: 9)
>> Name                :
>> RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size                : 2.728 TB
>> Sector Size         : 512
>> Is VD emulated      : Yes
>> Parity Size         : 0
>> State               : Optimal
>> Strip Size          : 64 KB
>> Number Of Drives    : 1
>> Span Depth          : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy   : Enabled
>> Encryption Type     : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 10 (Target Id: 10)
>> Name                :
>> RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size                : 2.728 TB
>> Sector Size         : 512
>> Is VD emulated      : Yes
>> Parity Size         : 0
>> State               : Optimal
>> Strip Size          : 64 KB
>> Number Of Drives    : 1
>> Span Depth          : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy   : Enabled
>> Encryption Type     : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 11 (Target Id: 11)
>> Name                :
>> RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size                : 2.728 TB
>> Sector Size         : 512
>> Is VD emulated      : Yes
>> Parity Size         : 0
>> State               : Optimal
>> Strip Size          : 64 KB
>> Number Of Drives    : 1
>> Span Depth          : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy   : Enabled
>> Encryption Type     : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 12 (Target Id: 12)
>> Name                :
>> RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size                : 2.728 TB
>> Sector Size         : 512
>> Is VD emulated      : Yes
>> Parity Size         : 0
>> State               : Optimal
>> Strip Size          : 64 KB
>> Number Of Drives    : 1
>> Span Depth          : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy   : Enabled
>> Encryption Type     : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 13 (Target Id: 13)
>> Name                :
>> RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size                : 2.728 TB
>> Sector Size         : 512
>> Is VD emulated      : Yes
>> Parity Size         : 0
>> State               : Optimal
>> Strip Size          : 64 KB
>> Number Of Drives    : 1
>> Span Depth          : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy   : Enabled
>> Encryption Type     : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 14 (Target Id: 14)
>> Name                :
>> RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size                : 2.728 TB
>> Sector Size         : 512
>> Is VD emulated      : Yes
>> Parity Size         : 0
>> State               : Optimal
>> Strip Size          : 64 KB
>> Number Of Drives    : 1
>> Span Depth          : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy   : Enabled
>> Encryption Type     : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>> Virtual Drive: 15 (Target Id: 15)
>> Name                :
>> RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
>> Size                : 2.728 TB
>> Sector Size         : 512
>> Is VD emulated      : Yes
>> Parity Size         : 0
>> State               : Optimal
>> Strip Size          : 256 KB
>> Number Of Drives    : 1
>> Span Depth          : 1
>> Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
>> Bad BBU
>> Default Access Policy: Read/Write
>> Current Access Policy: Read/Write
>> Disk Cache Policy   : Enabled
>> Encryption Type     : None
>> Bad Blocks Exist: No
>> Is VD Cached: No
>>
>>
>>
>> Exit Code: 0x00
>> ###############################################
>>                                      Adapter 0: Patrol Read Information:
>>
>> Patrol Read Mode: Auto
>> Patrol Read Execution Delay: 672 hours
>> Number of iterations completed: 2
>> Next start time: 12/13/2014, 03:00:00
>> Current State: Stopped
>> Patrol Read on SSD Devices: Disabled
>>
>> Exit Code: 0x00
>> ###############################################
>>                                      Check Consistency on VD #0 is
>> not in progress.
>> Check Consistency on VD #1 is not in progress.
>> Check Consistency on VD #2 is not in progress.
>> Check Consistency on VD #3 is not in progress.
>> Check Consistency on VD #4 is not in progress.
>> Check Consistency on VD #5 is not in progress.
>> Check Consistency on VD #6 is not in progress.
>> Check Consistency on VD #7 is not in progress.
>> Check Consistency on VD #8 is not in progress.
>> Check Consistency on VD #9 is not in progress.
>> Check Consistency on VD #10 is not in progress.
>> Check Consistency on VD #11 is not in progress.
>> Check Consistency on VD #12 is not in progress.
>> Check Consistency on VD #13 is not in progress.
>> Check Consistency on VD #14 is not in progress.
>> Check Consistency on VD #15 is not in progress.
>>
>> Exit Code: 0x00
>> root at colossus:~ # ls -l /dev/mfi*
>> crw-r-----  1 root  operator  0x22 Dec  5 17:18 /dev/mfi0
>> crw-r-----  1 root  operator  0x68 Dec  5 17:18 /dev/mfid0
>> crw-r-----  1 root  operator  0x69 Dec  5 17:18 /dev/mfid1
>> crw-r-----  1 root  operator  0x78 Dec  5 17:18 /dev/mfid10
>> crw-r-----  1 root  operator  0x79 Dec  5 17:18 /dev/mfid11
>> crw-r-----  1 root  operator  0x7a Dec  5 17:18 /dev/mfid12
>> crw-r-----  1 root  operator  0x82 Dec  5 17:18 /dev/mfid13
>> crw-r-----  1 root  operator  0x83 Dec  5 17:18 /dev/mfid14
>> crw-r-----  1 root  operator  0x84 Dec  5 17:18 /dev/mfid15
>> crw-r-----  1 root  operator  0x6a Dec  5 17:18 /dev/mfid2
>> crw-r-----  1 root  operator  0x6b Dec  5 17:18 /dev/mfid3
>> crw-r-----  1 root  operator  0x6c Dec  5 17:18 /dev/mfid4
>> crw-r-----  1 root  operator  0x6d Dec  5 17:18 /dev/mfid5
>> crw-r-----  1 root  operator  0x6e Dec  5 17:18 /dev/mfid6
>> crw-r-----  1 root  operator  0x75 Dec  5 17:18 /dev/mfid7
>> crw-r-----  1 root  operator  0x76 Dec  5 17:18 /dev/mfid8
>> crw-r-----  1 root  operator  0x77 Dec  5 17:18 /dev/mfid9
>> root at colossus:~ #
>>
>
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"


-- 
Michelle Sullivan
http://www.mhix.org/



More information about the freebsd-fs mailing list