ZFS 9.2/9.3 oh so broken...

Matthew Sullivan matthew at sorbs.net
Fri Sep 18 22:36:36 UTC 2015


Ok it's 9.2 so "unsupported" - but I believe the code is unchanged for
this one in 9.3.... which *is* supported....


Here's the last couple of days after the failure of a drive...  I
realised there was no point in swaping back in the new drive and I
should make the swap drive perm, and use the new drive as a hotspare....
so followed: http://docs.oracle.com/cd/E19253-01/819-5461/gcvdi/index.html

... that was a mistake...


[root at colossus ~]# praid ; zpool status -vx storage

MegaCli Tools Used to Gather Raid Info

==== Controllers
=======================================================================================================================
C#  Name                 CacheSize       FirmwareVer     BIOSver        
BBU            
0   lsi megaraid sas 9260-16i 512mb           2.130.403-3066  3.30.02.0
4.16.08.00 0x06060900 Missing        

---- BBUs
------------------------------------------------------------------------------------------------------------------------------
C#  Battery Type         Initialized     Voltage         Temperature    
Charge State    Alerts         
0                       
-                                                                            
 

---- Virtual Disks
---------------------------------------------------------------------------------------------------------------------
C#  Logical Volumes      Physical Disks  Degraded        Offline        
Critical        Failed         
0   16                   16              0               0              
0               0              

---- Virtual Disks Info
----------------------------------------------------------------------------------------------------------------
C#  ID                   Name            State           Size           
Raid Level     
0   L0                   mfid0           optimal         2.728 tb       
raid-0         
0   L1                                   optimal         2.728 tb       
raid-0         
0   L10                                  optimal         2.728 tb       
raid-0         
0   L11                                  optimal         2.728 tb       
raid-0         
0   L12                                  optimal         2.728 tb       
raid-0         
0   L13                                  optimal         2.728 tb       
raid-0         
0   L14                                  optimal         2.728 tb       
raid-0         
0   L15                                  optimal         2.728 tb       
raid-0         
0   L2                                   optimal         2.728 tb       
raid-0         
0   L3                                   optimal         2.728 tb       
raid-0         
0   L4                                   optimal         2.728 tb       
raid-0         
0   L5                                   optimal         2.728 tb       
raid-0         
0   L6                                   optimal         2.728 tb       
raid-0         
0   L7                                   optimal         2.728 tb       
raid-0         
0   L8                                   optimal         2.728 tb       
raid-0         
0   L9                                   optimal         2.728 tb       
raid-0         

---- Controller 0 Physical Drives
------------------------------------------------------------------------------------------------------
Enclosure: 245 (a0 => unavailable)
C#  Virtual Member       Slot   Model                    
Size            Serial          Media Err  Other Err 
Status               F-State  SAS Address      
0                        0      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320010
0                        1      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320011
0                        10     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320022
0                        11     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320023
0                        12     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320018
0                        13     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320019
0                        14     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001a
0                        15     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001b
0                        2      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320012
0                        3      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320013
0                        4      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000c
0                        5      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000d
0                        6      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000e
0                        7      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000f
0                        8      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320020
0                        9      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320021
  pool: storage
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Sep 17 00:20:00 2015
        9.20T scanned out of 27.1T at 83.2M/s, 62h34m to go
        625G resilvered, 33.97% done
config:

    NAME                          STATE     READ WRITE CKSUM
    storage                       DEGRADED     0     0     0
      raidz2-0                    DEGRADED     0     0     0
        mfid10                    ONLINE       0     0     0
        mfid8                     ONLINE       0     0     0
        mfid12                    ONLINE       0     0     0
        mfid0                     ONLINE       0     0     0
        mfid14                    ONLINE       0     0     0
        mfid15                    ONLINE       0     0     0
        mfid1                     ONLINE       0     0     0
        mfid7                     ONLINE       0     0     0
        mfid2                     ONLINE       0     0     0
        mfid9                     ONLINE       0     0     0
        mfid3                     ONLINE       0     0     0
        mfid4                     ONLINE       0     0     2
        mfid5                     ONLINE       0     0     0
        mfid6                     ONLINE       0     0     0
        spare-14                  DEGRADED     0     0     0
          replacing-0             DEGRADED     0     0     0
            15695558693726858796  UNAVAIL      0     0     0  was /dev/mfid7
            mfid13                ONLINE       0     0     0  (resilvering)
          mfid11                  ONLINE       0     0     0
    spares
      14948854088277424304        INUSE     was /dev/mfid11

errors: No known data errors
[root at colossus ~]# praid ; zpool status -vx storage

MegaCli Tools Used to Gather Raid Info

==== Controllers
=======================================================================================================================
C#  Name                 CacheSize       FirmwareVer     BIOSver        
BBU            
0   lsi megaraid sas 9260-16i 512mb           2.130.403-3066  3.30.02.0
4.16.08.00 0x06060900 Missing        

---- BBUs
------------------------------------------------------------------------------------------------------------------------------
C#  Battery Type         Initialized     Voltage         Temperature    
Charge State    Alerts         
0                       
-                                                                            
 

---- Virtual Disks
---------------------------------------------------------------------------------------------------------------------
C#  Logical Volumes      Physical Disks  Degraded        Offline        
Critical        Failed         
0   16                   16              0               0              
0               0              

---- Virtual Disks Info
----------------------------------------------------------------------------------------------------------------
C#  ID                   Name            State           Size           
Raid Level     
0   L0                   mfid0           optimal         2.728 tb       
raid-0         
0   L1                                   optimal         2.728 tb       
raid-0         
0   L10                                  optimal         2.728 tb       
raid-0         
0   L11                                  optimal         2.728 tb       
raid-0         
0   L12                                  optimal         2.728 tb       
raid-0         
0   L13                                  optimal         2.728 tb       
raid-0         
0   L14                                  optimal         2.728 tb       
raid-0         
0   L15                                  optimal         2.728 tb       
raid-0         
0   L2                                   optimal         2.728 tb       
raid-0         
0   L3                                   optimal         2.728 tb       
raid-0         
0   L4                                   optimal         2.728 tb       
raid-0         
0   L5                                   optimal         2.728 tb       
raid-0         
0   L6                                   optimal         2.728 tb       
raid-0         
0   L7                                   optimal         2.728 tb       
raid-0         
0   L8                                   optimal         2.728 tb       
raid-0         
0   L9                                   optimal         2.728 tb       
raid-0         

---- Controller 0 Physical Drives
------------------------------------------------------------------------------------------------------
Enclosure: 245 (a0 => unavailable)
C#  Virtual Member       Slot   Model                    
Size            Serial          Media Err  Other Err 
Status               F-State  SAS Address      
0                        0      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320010
0                        1      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320011
0                        10     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320022
0                        11     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320023
0                        12     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320018
0                        13     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320019
0                        14     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001a
0                        15     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001b
0                        2      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320012
0                        3      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320013
0                        4      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000c
0                        5      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000d
0                        6      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000e
0                        7      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000f
0                        8      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320020
0                        9      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320021
  pool: storage
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Sep 17 00:20:00 2015
        9.64T scanned out of 27.1T at 84.3M/s, 60h15m to go
        655G resilvered, 35.60% done
config:

    NAME                          STATE     READ WRITE CKSUM
    storage                       DEGRADED     0     0     0
      raidz2-0                    DEGRADED     0     0     0
        mfid10                    ONLINE       0     0     0
        mfid8                     ONLINE       0     0     0
        mfid12                    ONLINE       0     0     0
        mfid0                     ONLINE       0     0     0
        mfid14                    ONLINE       0     0     0
        mfid15                    ONLINE       0     0     0
        mfid1                     ONLINE       0     0     0
        mfid7                     ONLINE       0     0     0
        mfid2                     ONLINE       0     0     0
        mfid9                     ONLINE       0     0     0
        mfid3                     ONLINE       0     0     0
        mfid4                     ONLINE       0     0     2
        mfid5                     ONLINE       0     0     0
        mfid6                     ONLINE       0     0     0
        spare-14                  DEGRADED     0     0     0
          replacing-0             DEGRADED     0     0     0
            15695558693726858796  UNAVAIL      0     0     0  was /dev/mfid7
            mfid13                ONLINE       0     0     0  (resilvering)
          mfid11                  ONLINE       0     0     0
    spares
      14948854088277424304        INUSE     was /dev/mfid11

errors: No known data errors
You have new mail in /var/mail/root
[root at colossus ~]# praid ; zpool status -vx storage

MegaCli Tools Used to Gather Raid Info

==== Controllers
=======================================================================================================================
C#  Name                 CacheSize       FirmwareVer     BIOSver        
BBU            
0   lsi megaraid sas 9260-16i 512mb           2.130.403-3066  3.30.02.0
4.16.08.00 0x06060900 Missing        

---- BBUs
------------------------------------------------------------------------------------------------------------------------------
C#  Battery Type         Initialized     Voltage         Temperature    
Charge State    Alerts         
0                       
-                                                                            
 

---- Virtual Disks
---------------------------------------------------------------------------------------------------------------------
C#  Logical Volumes      Physical Disks  Degraded        Offline        
Critical        Failed         
0   16                   16              0               0              
0               0              

---- Virtual Disks Info
----------------------------------------------------------------------------------------------------------------
C#  ID                   Name            State           Size           
Raid Level     
0   L0                   mfid0           optimal         2.728 tb       
raid-0         
0   L1                                   optimal         2.728 tb       
raid-0         
0   L10                                  optimal         2.728 tb       
raid-0         
0   L11                                  optimal         2.728 tb       
raid-0         
0   L12                                  optimal         2.728 tb       
raid-0         
0   L13                                  optimal         2.728 tb       
raid-0         
0   L14                                  optimal         2.728 tb       
raid-0         
0   L15                                  optimal         2.728 tb       
raid-0         
0   L2                                   optimal         2.728 tb       
raid-0         
0   L3                                   optimal         2.728 tb       
raid-0         
0   L4                                   optimal         2.728 tb       
raid-0         
0   L5                                   optimal         2.728 tb       
raid-0         
0   L6                                   optimal         2.728 tb       
raid-0         
0   L7                                   optimal         2.728 tb       
raid-0         
0   L8                                   optimal         2.728 tb       
raid-0         
0   L9                                   optimal         2.728 tb       
raid-0         

---- Controller 0 Physical Drives
------------------------------------------------------------------------------------------------------
Enclosure: 245 (a0 => unavailable)
C#  Virtual Member       Slot   Model                    
Size            Serial          Media Err  Other Err 
Status               F-State  SAS Address      
0                        0      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320010
0                        1      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320011
0                        10     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320022
0                        11     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320023
0                        12     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320018
0                        13     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320019
0                        14     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001a
0                        15     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001b
0                        2      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320012
0                        3      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320013
0                        4      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000c
0                        5      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000d
0                        6      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000e
0                        7      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000f
0                        8      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320020
0                        9      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320021
  pool: storage
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Sep 17 00:20:00 2015
        11.8T scanned out of 27.1T at 87.5M/s, 50h46m to go
        804G resilvered, 43.69% done
config:

    NAME                          STATE     READ WRITE CKSUM
    storage                       DEGRADED     0     0     0
      raidz2-0                    DEGRADED     0     0     0
        mfid10                    ONLINE       0     0     0
        mfid8                     ONLINE       0     0     0
        mfid12                    ONLINE       0     0     0
        mfid0                     ONLINE       0     0     0
        mfid14                    ONLINE       0     0     0
        mfid15                    ONLINE       0     0     0
        mfid1                     ONLINE       0     0     0
        mfid7                     ONLINE       0     0     0
        mfid2                     ONLINE       0     0     0
        mfid9                     ONLINE       0     0     0
        mfid3                     ONLINE       0     0     0
        mfid4                     ONLINE       0     0     2
        mfid5                     ONLINE       0     0     0
        mfid6                     ONLINE       0     0     0
        spare-14                  DEGRADED     0     0     0
          replacing-0             DEGRADED     0     0     0
            15695558693726858796  UNAVAIL      0     0     0  was /dev/mfid7
            mfid13                ONLINE       0     0     0  (resilvering)
          mfid11                  ONLINE       0     0     0
    spares
      14948854088277424304        INUSE     was /dev/mfid11

errors: No known data errors
You have new mail in /var/mail/root
[root at colossus ~]# praid ; zpool status -vx storage

MegaCli Tools Used to Gather Raid Info

==== Controllers
=======================================================================================================================
C#  Name                 CacheSize       FirmwareVer     BIOSver        
BBU            
0   lsi megaraid sas 9260-16i 512mb           2.130.403-3066  3.30.02.0
4.16.08.00 0x06060900 Missing        

---- BBUs
------------------------------------------------------------------------------------------------------------------------------
C#  Battery Type         Initialized     Voltage         Temperature    
Charge State    Alerts         
0                       
-                                                                            
 

---- Virtual Disks
---------------------------------------------------------------------------------------------------------------------
C#  Logical Volumes      Physical Disks  Degraded        Offline        
Critical        Failed         
0   16                   16              0               0              
0               0              

---- Virtual Disks Info
----------------------------------------------------------------------------------------------------------------
C#  ID                   Name            State           Size           
Raid Level     
0   L0                   mfid0           optimal         2.728 tb       
raid-0         
0   L1                                   optimal         2.728 tb       
raid-0         
0   L10                                  optimal         2.728 tb       
raid-0         
0   L11                                  optimal         2.728 tb       
raid-0         
0   L12                                  optimal         2.728 tb       
raid-0         
0   L13                                  optimal         2.728 tb       
raid-0         
0   L14                                  optimal         2.728 tb       
raid-0         
0   L15                                  optimal         2.728 tb       
raid-0         
0   L2                                   optimal         2.728 tb       
raid-0         
0   L3                                   optimal         2.728 tb       
raid-0         
0   L4                                   optimal         2.728 tb       
raid-0         
0   L5                                   optimal         2.728 tb       
raid-0         
0   L6                                   optimal         2.728 tb       
raid-0         
0   L7                                   optimal         2.728 tb       
raid-0         
0   L8                                   optimal         2.728 tb       
raid-0         
0   L9                                   optimal         2.728 tb       
raid-0         

---- Controller 0 Physical Drives
------------------------------------------------------------------------------------------------------
Enclosure: 245 (a0 => unavailable)
C#  Virtual Member       Slot   Model                    
Size            Serial          Media Err  Other Err 
Status               F-State  SAS Address      
0                        0      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320010
0                        1      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320011
0                        10     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320022
0                        11     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320023
0                        12     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320018
0                        13     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320019
0                        14     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001a
0                        15     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001b
0                        2      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320012
0                        3      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320013
0                        4      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000c
0                        5      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000d
0                        6      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000e
0                        7      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000f
0                        8      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320020
0                        9      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320021
  pool: storage
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Sep 17 00:20:00 2015
        12.0T scanned out of 27.1T at 87.8M/s, 50h0m to go
        816G resilvered, 44.35% done
config:

    NAME                          STATE     READ WRITE CKSUM
    storage                       DEGRADED     0     0     0
      raidz2-0                    DEGRADED     0     0     0
        mfid10                    ONLINE       0     0     0
        mfid8                     ONLINE       0     0     0
        mfid12                    ONLINE       0     0     0
        mfid0                     ONLINE       0     0     0
        mfid14                    ONLINE       0     0     0
        mfid15                    ONLINE       0     0     0
        mfid1                     ONLINE       0     0     0
        mfid7                     ONLINE       0     0     0
        mfid2                     ONLINE       0     0     0
        mfid9                     ONLINE       0     0     0
        mfid3                     ONLINE       0     0     0
        mfid4                     ONLINE       0     0     2
        mfid5                     ONLINE       0     0     0
        mfid6                     ONLINE       0     0     0
        spare-14                  DEGRADED     0     0     0
          replacing-0             DEGRADED     0     0     0
            15695558693726858796  UNAVAIL      0     0     0  was /dev/mfid7
            mfid13                ONLINE       0     0     0  (resilvering)
          mfid11                  ONLINE       0     0     0
    spares
      14948854088277424304        INUSE     was /dev/mfid11

errors: No known data errors
You have new mail in /var/mail/root
[root at colossus ~]# praid ; zpool status -vx storage

MegaCli Tools Used to Gather Raid Info

==== Controllers
=======================================================================================================================
C#  Name                 CacheSize       FirmwareVer     BIOSver        
BBU            
0   lsi megaraid sas 9260-16i 512mb           2.130.403-3066  3.30.02.0
4.16.08.00 0x06060900 Missing        

---- BBUs
------------------------------------------------------------------------------------------------------------------------------
C#  Battery Type         Initialized     Voltage         Temperature    
Charge State    Alerts         
0                       
-                                                                            
 

---- Virtual Disks
---------------------------------------------------------------------------------------------------------------------
C#  Logical Volumes      Physical Disks  Degraded        Offline        
Critical        Failed         
0   16                   16              0               0              
0               0              

---- Virtual Disks Info
----------------------------------------------------------------------------------------------------------------
C#  ID                   Name            State           Size           
Raid Level     
0   L0                   mfid0           optimal         2.728 tb       
raid-0         
0   L1                                   optimal         2.728 tb       
raid-0         
0   L10                                  optimal         2.728 tb       
raid-0         
0   L11                                  optimal         2.728 tb       
raid-0         
0   L12                                  optimal         2.728 tb       
raid-0         
0   L13                                  optimal         2.728 tb       
raid-0         
0   L14                                  optimal         2.728 tb       
raid-0         
0   L15                                  optimal         2.728 tb       
raid-0         
0   L2                                   optimal         2.728 tb       
raid-0         
0   L3                                   optimal         2.728 tb       
raid-0         
0   L4                                   optimal         2.728 tb       
raid-0         
0   L5                                   optimal         2.728 tb       
raid-0         
0   L6                                   optimal         2.728 tb       
raid-0         
0   L7                                   optimal         2.728 tb       
raid-0         
0   L8                                   optimal         2.728 tb       
raid-0         
0   L9                                   optimal         2.728 tb       
raid-0         

---- Controller 0 Physical Drives
------------------------------------------------------------------------------------------------------
Enclosure: 245 (a0 => unavailable)
C#  Virtual Member       Slot   Model                    
Size            Serial          Media Err  Other Err 
Status               F-State  SAS Address      
0                        0      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320010
0                        1      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320011
0                        10     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320022
0                        11     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320023
0                        12     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320018
0                        13     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320019
0                        14     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001a
0                        15     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001b
0                        2      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320012
0                        3      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320013
0                        4      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000c
0                        5      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000d
0                        6      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000e
0                        7      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000f
0                        8      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320020
0                        9      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320021
  pool: storage
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Sep 17 00:20:00 2015
        12.0T scanned out of 27.1T at 87.7M/s, 49h54m to go
        819G resilvered, 44.49% done
config:

    NAME                          STATE     READ WRITE CKSUM
    storage                       DEGRADED     0     0     0
      raidz2-0                    DEGRADED     0     0     0
        mfid10                    ONLINE       0     0     0
        mfid8                     ONLINE       0     0     0
        mfid12                    ONLINE       0     0     0
        mfid0                     ONLINE       0     0     0
        mfid14                    ONLINE       0     0     0
        mfid15                    ONLINE       0     0     0
        mfid1                     ONLINE       0     0     0
        mfid7                     ONLINE       0     0     0
        mfid2                     ONLINE       0     0     0
        mfid9                     ONLINE       0     0     0
        mfid3                     ONLINE       0     0     0
        mfid4                     ONLINE       0     0     2
        mfid5                     ONLINE       0     0     0
        mfid6                     ONLINE       0     0     0
        spare-14                  DEGRADED     0     0     0
          replacing-0             DEGRADED     0     0     0
            15695558693726858796  UNAVAIL      0     0     0  was /dev/mfid7
            mfid13                ONLINE       0     0     0  (resilvering)
          mfid11                  ONLINE       0     0     0
    spares
      14948854088277424304        INUSE     was /dev/mfid11

errors: No known data errors
[root at colossus ~]# praid ; zpool status -vx storage

MegaCli Tools Used to Gather Raid Info

==== Controllers
=======================================================================================================================
C#  Name                 CacheSize       FirmwareVer     BIOSver        
BBU            
0   lsi megaraid sas 9260-16i 512mb           2.130.403-3066  3.30.02.0
4.16.08.00 0x06060900 Missing        

---- BBUs
------------------------------------------------------------------------------------------------------------------------------
C#  Battery Type         Initialized     Voltage         Temperature    
Charge State    Alerts         
0                       
-                                                                            
 

---- Virtual Disks
---------------------------------------------------------------------------------------------------------------------
C#  Logical Volumes      Physical Disks  Degraded        Offline        
Critical        Failed         
0   16                   16              0               0              
0               0              

---- Virtual Disks Info
----------------------------------------------------------------------------------------------------------------
C#  ID                   Name            State           Size           
Raid Level     
0   L0                   mfid0           optimal         2.728 tb       
raid-0         
0   L1                                   optimal         2.728 tb       
raid-0         
0   L10                                  optimal         2.728 tb       
raid-0         
0   L11                                  optimal         2.728 tb       
raid-0         
0   L12                                  optimal         2.728 tb       
raid-0         
0   L13                                  optimal         2.728 tb       
raid-0         
0   L14                                  optimal         2.728 tb       
raid-0         
0   L15                                  optimal         2.728 tb       
raid-0         
0   L2                                   optimal         2.728 tb       
raid-0         
0   L3                                   optimal         2.728 tb       
raid-0         
0   L4                                   optimal         2.728 tb       
raid-0         
0   L5                                   optimal         2.728 tb       
raid-0         
0   L6                                   optimal         2.728 tb       
raid-0         
0   L7                                   optimal         2.728 tb       
raid-0         
0   L8                                   optimal         2.728 tb       
raid-0         
0   L9                                   optimal         2.728 tb       
raid-0         

---- Controller 0 Physical Drives
------------------------------------------------------------------------------------------------------
Enclosure: 245 (a0 => unavailable)
C#  Virtual Member       Slot   Model                    
Size            Serial          Media Err  Other Err 
Status               F-State  SAS Address      
0                        0      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320010
0                        1      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320011
0                        10     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320022
0                        11     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320023
0                        12     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320018
0                        13     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320019
0                        14     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001a
0                        15     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001b
0                        2      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320012
0                        3      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320013
0                        4      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000c
0                        5      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000d
0                        6      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000e
0                        7      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000f
0                        8      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320020
0                        9      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320021
  pool: storage
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Sep 17 00:20:00 2015
        13.7T scanned out of 27.1T at 87.3M/s, 44h27m to go
        934G resilvered, 50.77% done
config:

    NAME                          STATE     READ WRITE CKSUM
    storage                       DEGRADED     0     0     0
      raidz2-0                    DEGRADED     0     0     0
        mfid10                    ONLINE       0     0     0
        mfid8                     ONLINE       0     0     0
        mfid12                    ONLINE       0     0     0
        mfid0                     ONLINE       0     0     0
        mfid14                    ONLINE       0     0     0
        mfid15                    ONLINE       0     0     0
        mfid1                     ONLINE       0     0     0
        mfid7                     ONLINE       0     0     0
        mfid2                     ONLINE       0     0     0
        mfid9                     ONLINE       0     0     0
        mfid3                     ONLINE       0     0     0
        mfid4                     ONLINE       0     0     2
        mfid5                     ONLINE       0     0     0
        mfid6                     ONLINE       0     0     0
        spare-14                  DEGRADED     0     0     0
          replacing-0             DEGRADED     0     0     0
            15695558693726858796  UNAVAIL      0     0     0  was /dev/mfid7
            mfid13                ONLINE       0     0     0  (resilvering)
          mfid11                  ONLINE       0     0     0
    spares
      14948854088277424304        INUSE     was /dev/mfid11

errors: No known data errors
You have new mail in /var/mail/root
[root at colossus ~]# zpool add storage spare mfid13
invalid vdev specification
use '-f' to override the following errors:
/dev/mfid13 is part of active pool 'storage'
[root at colossus ~]# praid ; zpool status -vx storage

MegaCli Tools Used to Gather Raid Info

==== Controllers
=======================================================================================================================
C#  Name                 CacheSize       FirmwareVer     BIOSver        
BBU            
0   lsi megaraid sas 9260-16i 512mb           2.130.403-3066  3.30.02.0
4.16.08.00 0x06060900 Missing        

---- BBUs
------------------------------------------------------------------------------------------------------------------------------
C#  Battery Type         Initialized     Voltage         Temperature    
Charge State    Alerts         
0                       
-                                                                            
 

---- Virtual Disks
---------------------------------------------------------------------------------------------------------------------
C#  Logical Volumes      Physical Disks  Degraded        Offline        
Critical        Failed         
0   16                   16              0               0              
0               0              

---- Virtual Disks Info
----------------------------------------------------------------------------------------------------------------
C#  ID                   Name            State           Size           
Raid Level     
0   L0                   mfid0           optimal         2.728 tb       
raid-0         
0   L1                                   optimal         2.728 tb       
raid-0         
0   L10                                  optimal         2.728 tb       
raid-0         
0   L11                                  optimal         2.728 tb       
raid-0         
0   L12                                  optimal         2.728 tb       
raid-0         
0   L13                                  optimal         2.728 tb       
raid-0         
0   L14                                  optimal         2.728 tb       
raid-0         
0   L15                                  optimal         2.728 tb       
raid-0         
0   L2                                   optimal         2.728 tb       
raid-0         
0   L3                                   optimal         2.728 tb       
raid-0         
0   L4                                   optimal         2.728 tb       
raid-0         
0   L5                                   optimal         2.728 tb       
raid-0         
0   L6                                   optimal         2.728 tb       
raid-0         
0   L7                                   optimal         2.728 tb       
raid-0         
0   L8                                   optimal         2.728 tb       
raid-0         
0   L9                                   optimal         2.728 tb       
raid-0         

---- Controller 0 Physical Drives
------------------------------------------------------------------------------------------------------
Enclosure: 245 (a0 => unavailable)
C#  Virtual Member       Slot   Model                    
Size            Serial          Media Err  Other Err 
Status               F-State  SAS Address      
0                        0      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320010
0                        1      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320011
0                        10     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320022
0                        11     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320023
0                        12     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320018
0                        13     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320019
0                        14     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001a
0                        15     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001b
0                        2      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320012
0                        3      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320013
0                        4      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000c
0                        5      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000d
0                        6      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000e
0                        7      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000f
0                        8      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320020
0                        9      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320021
  pool: storage
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Sep 17 00:20:00 2015
        13.8T scanned out of 27.1T at 87.3M/s, 44h24m to go
        935G resilvered, 50.83% done
config:

    NAME                          STATE     READ WRITE CKSUM
    storage                       DEGRADED     0     0     0
      raidz2-0                    DEGRADED     0     0     0
        mfid10                    ONLINE       0     0     0
        mfid8                     ONLINE       0     0     0
        mfid12                    ONLINE       0     0     0
        mfid0                     ONLINE       0     0     0
        mfid14                    ONLINE       0     0     0
        mfid15                    ONLINE       0     0     0
        mfid1                     ONLINE       0     0     0
        mfid7                     ONLINE       0     0     0
        mfid2                     ONLINE       0     0     0
        mfid9                     ONLINE       0     0     0
        mfid3                     ONLINE       0     0     0
        mfid4                     ONLINE       0     0     2
        mfid5                     ONLINE       0     0     0
        mfid6                     ONLINE       0     0     0
        spare-14                  DEGRADED     0     0     0
          replacing-0             DEGRADED     0     0     0
            15695558693726858796  UNAVAIL      0     0     0  was /dev/mfid7
            mfid13                ONLINE       0     0     0  (resilvering)
          mfid11                  ONLINE       0     0     0
    spares
      14948854088277424304        INUSE     was /dev/mfid11

errors: No known data errors
[root at colossus ~]# zpool detach storage mfid13
[root at colossus ~]# zpool add storage spare mfid13
praid ; zpool status -vx storage
[root at colossus ~]# praid ; zpool status -vx storage

MegaCli Tools Used to Gather Raid Info

==== Controllers
=======================================================================================================================
C#  Name                 CacheSize       FirmwareVer     BIOSver        
BBU            
0   lsi megaraid sas 9260-16i 512mb           2.130.403-3066  3.30.02.0
4.16.08.00 0x06060900 Missing        

---- BBUs
------------------------------------------------------------------------------------------------------------------------------
C#  Battery Type         Initialized     Voltage         Temperature    
Charge State    Alerts         
0                       
-                                                                            
 

---- Virtual Disks
---------------------------------------------------------------------------------------------------------------------
C#  Logical Volumes      Physical Disks  Degraded        Offline        
Critical        Failed         
0   16                   16              0               0              
0               0              

---- Virtual Disks Info
----------------------------------------------------------------------------------------------------------------
C#  ID                   Name            State           Size           
Raid Level     
0   L0                   mfid0           optimal         2.728 tb       
raid-0         
0   L1                                   optimal         2.728 tb       
raid-0         
0   L10                                  optimal         2.728 tb       
raid-0         
0   L11                                  optimal         2.728 tb       
raid-0         
0   L12                                  optimal         2.728 tb       
raid-0         
0   L13                                  optimal         2.728 tb       
raid-0         
0   L14                                  optimal         2.728 tb       
raid-0         
0   L15                                  optimal         2.728 tb       
raid-0         
0   L2                                   optimal         2.728 tb       
raid-0         
0   L3                                   optimal         2.728 tb       
raid-0         
0   L4                                   optimal         2.728 tb       
raid-0         
0   L5                                   optimal         2.728 tb       
raid-0         
0   L6                                   optimal         2.728 tb       
raid-0         
0   L7                                   optimal         2.728 tb       
raid-0         
0   L8                                   optimal         2.728 tb       
raid-0         
0   L9                                   optimal         2.728 tb       
raid-0         

---- Controller 0 Physical Drives
------------------------------------------------------------------------------------------------------
Enclosure: 245 (a0 => unavailable)
C#  Virtual Member       Slot   Model                    
Size            Serial          Media Err  Other Err 
Status               F-State  SAS Address      
0                        0      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320010
0                        1      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320011
0                        10     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320022
0                        11     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320023
0                        12     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320018
0                        13     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320019
0                        14     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001a
0                        15     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001b
0                        2      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320012
0                        3      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320013
0                        4      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000c
0                        5      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000d
0                        6      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000e
0                        7      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000f
0                        8      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320020
0                        9      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320021
  pool: storage
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Sep 17 00:20:00 2015
        13.8T scanned out of 27.1T at 87.5M/s, 44h16m to go
        935G resilvered, 50.92% done
config:

    NAME                        STATE     READ WRITE CKSUM
    storage                     DEGRADED     0     0     0
      raidz2-0                  DEGRADED     0     0     0
        mfid10                  ONLINE       0     0     0
        mfid8                   ONLINE       0     0     0
        mfid12                  ONLINE       0     0     0
        mfid0                   ONLINE       0     0     0
        mfid14                  ONLINE       0     0     0
        mfid15                  ONLINE       0     0     0
        mfid1                   ONLINE       0     0     0
        mfid7                   ONLINE       0     0     0
        mfid2                   ONLINE       0     0     0
        mfid9                   ONLINE       0     0     0
        mfid3                   ONLINE       0     0     0
        mfid4                   ONLINE       0     0     2
        mfid5                   ONLINE       0     0     0
        mfid6                   ONLINE       0     0     0
        spare-14                DEGRADED     0     0     0
          15695558693726858796  UNAVAIL      0     0     0  was /dev/mfid7
          mfid11                ONLINE       0     0     0
    spares
      14948854088277424304      INUSE     was /dev/mfid11
      mfid13                    AVAIL   

errors: No known data errors
[root at colossus ~]# zpool detach storage 15695558693726858796
[root at colossus ~]# praid ; zpool status -vx storage

MegaCli Tools Used to Gather Raid Info

==== Controllers
=======================================================================================================================
C#  Name                 CacheSize       FirmwareVer     BIOSver        
BBU            
0   lsi megaraid sas 9260-16i 512mb           2.130.403-3066  3.30.02.0
4.16.08.00 0x06060900 Missing        

---- BBUs
------------------------------------------------------------------------------------------------------------------------------
C#  Battery Type         Initialized     Voltage         Temperature    
Charge State    Alerts         
0                       
-                                                                            
 

---- Virtual Disks
---------------------------------------------------------------------------------------------------------------------
C#  Logical Volumes      Physical Disks  Degraded        Offline        
Critical        Failed         
0   16                   16              0               0              
0               0              

---- Virtual Disks Info
----------------------------------------------------------------------------------------------------------------
C#  ID                   Name            State           Size           
Raid Level     
0   L0                   mfid0           optimal         2.728 tb       
raid-0         
0   L1                                   optimal         2.728 tb       
raid-0         
0   L10                                  optimal         2.728 tb       
raid-0         
0   L11                                  optimal         2.728 tb       
raid-0         
0   L12                                  optimal         2.728 tb       
raid-0         
0   L13                                  optimal         2.728 tb       
raid-0         
0   L14                                  optimal         2.728 tb       
raid-0         
0   L15                                  optimal         2.728 tb       
raid-0         
0   L2                                   optimal         2.728 tb       
raid-0         
0   L3                                   optimal         2.728 tb       
raid-0         
0   L4                                   optimal         2.728 tb       
raid-0         
0   L5                                   optimal         2.728 tb       
raid-0         
0   L6                                   optimal         2.728 tb       
raid-0         
0   L7                                   optimal         2.728 tb       
raid-0         
0   L8                                   optimal         2.728 tb       
raid-0         
0   L9                                   optimal         2.728 tb       
raid-0         

---- Controller 0 Physical Drives
------------------------------------------------------------------------------------------------------
Enclosure: 245 (a0 => unavailable)
C#  Virtual Member       Slot   Model                    
Size            Serial          Media Err  Other Err 
Status               F-State  SAS Address      
0                        0      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320010
0                        1      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320011
0                        10     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320022
0                        11     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320023
0                        12     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320018
0                        13     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320019
0                        14     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001a
0                        15     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001b
0                        2      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320012
0                        3      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320013
0                        4      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000c
0                        5      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000d
0                        6      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000e
0                        7      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000f
0                        8      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320020
0                        9      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320021
  pool: storage
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Sep 17 00:20:00 2015
        13.8T scanned out of 27.1T at 87.6M/s, 44h5m to go
        935G resilvered, 51.03% done
config:

    NAME        STATE     READ WRITE CKSUM
    storage     ONLINE       0     0     0
      raidz2-0  ONLINE       0     0     0
        mfid10  ONLINE       0     0     0
        mfid8   ONLINE       0     0     0
        mfid12  ONLINE       0     0     0
        mfid0   ONLINE       0     0     0
        mfid14  ONLINE       0     0     0
        mfid15  ONLINE       0     0     0
        mfid1   ONLINE       0     0     0
        mfid7   ONLINE       0     0     0
        mfid2   ONLINE       0     0     0
        mfid9   ONLINE       0     0     0
        mfid3   ONLINE       0     0     0
        mfid4   ONLINE       0     0     2
        mfid5   ONLINE       0     0     0
        mfid6   ONLINE       0     0     0
        mfid11  ONLINE       0     0     0
    spares
      mfid13    AVAIL   

errors: No known data errors
[root at colossus ~]# zpool scrub storage
cannot scrub storage: currently resilvering
[root at colossus ~]# praid ; zpool status -vx storage

MegaCli Tools Used to Gather Raid Info

==== Controllers
=======================================================================================================================
C#  Name                 CacheSize       FirmwareVer     BIOSver        
BBU            
0   lsi megaraid sas 9260-16i 512mb           2.130.403-3066  3.30.02.0
4.16.08.00 0x06060900 Missing        

---- BBUs
------------------------------------------------------------------------------------------------------------------------------
C#  Battery Type         Initialized     Voltage         Temperature    
Charge State    Alerts         
0                       
-                                                                            
 

---- Virtual Disks
---------------------------------------------------------------------------------------------------------------------
C#  Logical Volumes      Physical Disks  Degraded        Offline        
Critical        Failed         
0   16                   16              0               0              
0               0              

---- Virtual Disks Info
----------------------------------------------------------------------------------------------------------------
C#  ID                   Name            State           Size           
Raid Level     
0   L0                   mfid0           optimal         2.728 tb       
raid-0         
0   L1                                   optimal         2.728 tb       
raid-0         
0   L10                                  optimal         2.728 tb       
raid-0         
0   L11                                  optimal         2.728 tb       
raid-0         
0   L12                                  optimal         2.728 tb       
raid-0         
0   L13                                  optimal         2.728 tb       
raid-0         
0   L14                                  optimal         2.728 tb       
raid-0         
0   L15                                  optimal         2.728 tb       
raid-0         
0   L2                                   optimal         2.728 tb       
raid-0         
0   L3                                   optimal         2.728 tb       
raid-0         
0   L4                                   optimal         2.728 tb       
raid-0         
0   L5                                   optimal         2.728 tb       
raid-0         
0   L6                                   optimal         2.728 tb       
raid-0         
0   L7                                   optimal         2.728 tb       
raid-0         
0   L8                                   optimal         2.728 tb       
raid-0         
0   L9                                   optimal         2.728 tb       
raid-0         

---- Controller 0 Physical Drives
------------------------------------------------------------------------------------------------------
Enclosure: 245 (a0 => unavailable)
C#  Virtual Member       Slot   Model                    
Size            Serial          Media Err  Other Err 
Status               F-State  SAS Address      
0                        0      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320010
0                        1      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320011
0                        10     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320022
0                        11     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320023
0                        12     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320018
0                        13     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320019
0                        14     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001a
0                        15     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001b
0                        2      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320012
0                        3      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320013
0                        4      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000c
0                        5      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000d
0                        6      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000e
0                        7      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000f
0                        8      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320020
0                        9      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320021
  pool: storage
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Sep 17 00:20:00 2015
        13.8T scanned out of 27.1T at 87.7M/s, 43h59m to go
        935G resilvered, 51.09% done
config:

    NAME        STATE     READ WRITE CKSUM
    storage     ONLINE       0     0     0
      raidz2-0  ONLINE       0     0     0
        mfid10  ONLINE       0     0     0
        mfid8   ONLINE       0     0     0
        mfid12  ONLINE       0     0     0
        mfid0   ONLINE       0     0     0
        mfid14  ONLINE       0     0     0
        mfid15  ONLINE       0     0     0
        mfid1   ONLINE       0     0     0
        mfid7   ONLINE       0     0     0
        mfid2   ONLINE       0     0     0
        mfid9   ONLINE       0     0     0
        mfid3   ONLINE       0     0     0
        mfid4   ONLINE       0     0     2
        mfid5   ONLINE       0     0     0
        mfid6   ONLINE       0     0     0
        mfid11  ONLINE       0     0     0
    spares
      mfid13    AVAIL   

errors: No known data errors
[root at colossus ~]# reboot
.
.
.
.
[michelle at colossus /usr/home/michelle]$ sudo su -
Password:
root at colossus:~ # praid ; zpool status -vx storage

MegaCli Tools Used to Gather Raid Info

==== Controllers
=======================================================================================================================
C#  Name                 CacheSize       FirmwareVer     BIOSver        
BBU            
0   lsi megaraid sas 9260-16i 512mb           2.130.403-3066  3.30.02.0
4.16.08.00 0x06060900 Missing        

---- BBUs
------------------------------------------------------------------------------------------------------------------------------
C#  Battery Type         Initialized     Voltage         Temperature    
Charge State    Alerts         
0                       
-                                                                            
 

---- Virtual Disks
---------------------------------------------------------------------------------------------------------------------
C#  Logical Volumes      Physical Disks  Degraded        Offline        
Critical        Failed         
0   16                   16              0               0              
0               0              

---- Virtual Disks Info
----------------------------------------------------------------------------------------------------------------
C#  ID                   Name            State           Size           
Raid Level     
0   L0                   mfid0           optimal         2.728 tb       
raid-0         
0   L1                                   optimal         2.728 tb       
raid-0         
0   L10                                  optimal         2.728 tb       
raid-0         
0   L11                                  optimal         2.728 tb       
raid-0         
0   L12                                  optimal         2.728 tb       
raid-0         
0   L13                                  optimal         2.728 tb       
raid-0         
0   L14                                  optimal         2.728 tb       
raid-0         
0   L15                                  optimal         2.728 tb       
raid-0         
0   L2                                   optimal         2.728 tb       
raid-0         
0   L3                                   optimal         2.728 tb       
raid-0         
0   L4                                   optimal         2.728 tb       
raid-0         
0   L5                                   optimal         2.728 tb       
raid-0         
0   L6                                   optimal         2.728 tb       
raid-0         
0   L7                                   optimal         2.728 tb       
raid-0         
0   L8                                   optimal         2.728 tb       
raid-0         
0   L9                                   optimal         2.728 tb       
raid-0         

---- Controller 0 Physical Drives
------------------------------------------------------------------------------------------------------
Enclosure: 245 (a0 => unavailable)
C#  Virtual Member       Slot   Model                    
Size            Serial          Media Err  Other Err 
Status               F-State  SAS Address      
0                        0      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320010
0                        1      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320011
0                        10     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320022
0                        11     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320023
0                        12     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320018
0                        13     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320019
0                        14     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001a
0                        15     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001b
0                        2      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320012
0                        3      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320013
0                        4      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000c
0                        5      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000d
0                        6      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000e
0                        7      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000f
0                        8      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320020
0                        9      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320021
  pool: storage
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Sep 17 00:20:00 2015
        13.9T scanned out of 27.1T at 1/s, (scan is slow, no estimated time)
        935G resilvered, 51.25% done
config:

    NAME        STATE     READ WRITE CKSUM
    storage     ONLINE       0     0     0
      raidz2-0  ONLINE       0     0     0
        mfid10  ONLINE       0     0     0
        mfid8   ONLINE       0     0     0
        mfid12  ONLINE       0     0     0
        mfid0   ONLINE       0     0     0
        mfid14  ONLINE       0     0     0
        mfid15  ONLINE       0     0     0
        mfid1   ONLINE       0     0     0
        mfid7   ONLINE       0     0     0
        mfid2   ONLINE       0     0     0
        mfid9   ONLINE       0     0     0
        mfid3   ONLINE       0     0     0
        mfid4   ONLINE       0     0     0
        mfid5   ONLINE       0     0     0
        mfid6   ONLINE       0     0     0
        mfid11  ONLINE       0     0     0
    spares
      mfid13    AVAIL   

errors: No known data errors
root at colossus:~ # zpool scrub storage
cannot scrub storage: currently resilvering
root at colossus:~ # praid ; zpool status -vx storage

MegaCli Tools Used to Gather Raid Info

==== Controllers
=======================================================================================================================
C#  Name                 CacheSize       FirmwareVer     BIOSver        
BBU            
0   lsi megaraid sas 9260-16i 512mb           2.130.403-3066  3.30.02.0
4.16.08.00 0x06060900 Missing        

---- BBUs
------------------------------------------------------------------------------------------------------------------------------
C#  Battery Type         Initialized     Voltage         Temperature    
Charge State    Alerts         
0                       
-                                                                            
 

---- Virtual Disks
---------------------------------------------------------------------------------------------------------------------
C#  Logical Volumes      Physical Disks  Degraded        Offline        
Critical        Failed         
0   16                   16              0               0              
0               0              

---- Virtual Disks Info
----------------------------------------------------------------------------------------------------------------
C#  ID                   Name            State           Size           
Raid Level     
0   L0                   mfid0           optimal         2.728 tb       
raid-0         
0   L1                                   optimal         2.728 tb       
raid-0         
0   L10                                  optimal         2.728 tb       
raid-0         
0   L11                                  optimal         2.728 tb       
raid-0         
0   L12                                  optimal         2.728 tb       
raid-0         
0   L13                                  optimal         2.728 tb       
raid-0         
0   L14                                  optimal         2.728 tb       
raid-0         
0   L15                                  optimal         2.728 tb       
raid-0         
0   L2                                   optimal         2.728 tb       
raid-0         
0   L3                                   optimal         2.728 tb       
raid-0         
0   L4                                   optimal         2.728 tb       
raid-0         
0   L5                                   optimal         2.728 tb       
raid-0         
0   L6                                   optimal         2.728 tb       
raid-0         
0   L7                                   optimal         2.728 tb       
raid-0         
0   L8                                   optimal         2.728 tb       
raid-0         
0   L9                                   optimal         2.728 tb       
raid-0         

---- Controller 0 Physical Drives
------------------------------------------------------------------------------------------------------
Enclosure: 245 (a0 => unavailable)
C#  Virtual Member       Slot   Model                    
Size            Serial          Media Err  Other Err 
Status               F-State  SAS Address      
0                        0      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320010
0                        1      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320011
0                        10     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320022
0                        11     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320023
0                        12     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320018
0                        13     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320019
0                        14     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001a
0                        15     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001b
0                        2      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320012
0                        3      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320013
0                        4      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000c
0                        5      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000d
0                        6      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000e
0                        7      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000f
0                        8      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320020
0                        9      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320021
  pool: storage
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Sep 17 00:20:00 2015
        13.9T scanned out of 27.1T at 1/s, (scan is slow, no estimated time)
        935G resilvered, 51.47% done
config:

    NAME        STATE     READ WRITE CKSUM
    storage     ONLINE       0     0     0
      raidz2-0  ONLINE       0     0     0
        mfid10  ONLINE       0     0     0
        mfid8   ONLINE       0     0     0
        mfid12  ONLINE       0     0     0
        mfid0   ONLINE       0     0     0
        mfid14  ONLINE       0     0     0
        mfid15  ONLINE       0     0     0
        mfid1   ONLINE       0     0     0
        mfid7   ONLINE       0     0     0
        mfid2   ONLINE       0     0     0
        mfid9   ONLINE       0     0     0
        mfid3   ONLINE       0     0     0
        mfid4   ONLINE       0     0     0
        mfid5   ONLINE       0     0     0
        mfid6   ONLINE       0     0     0
        mfid11  ONLINE       0     0     0
    spares
      mfid13    AVAIL   

errors: No known data errors
root at colossus:~ # praid ; zpool status -vx storage

MegaCli Tools Used to Gather Raid Info

==== Controllers
=======================================================================================================================
C#  Name                 CacheSize       FirmwareVer     BIOSver        
BBU            
0   lsi megaraid sas 9260-16i 512mb           2.130.403-3066  3.30.02.0
4.16.08.00 0x06060900 Missing        

---- BBUs
------------------------------------------------------------------------------------------------------------------------------
C#  Battery Type         Initialized     Voltage         Temperature    
Charge State    Alerts         
0                       
-                                                                            
 

---- Virtual Disks
---------------------------------------------------------------------------------------------------------------------
C#  Logical Volumes      Physical Disks  Degraded        Offline        
Critical        Failed         
0   16                   16              0               0              
0               0              

---- Virtual Disks Info
----------------------------------------------------------------------------------------------------------------
C#  ID                   Name            State           Size           
Raid Level     
0   L0                   mfid0           optimal         2.728 tb       
raid-0         
0   L1                                   optimal         2.728 tb       
raid-0         
0   L10                                  optimal         2.728 tb       
raid-0         
0   L11                                  optimal         2.728 tb       
raid-0         
0   L12                                  optimal         2.728 tb       
raid-0         
0   L13                                  optimal         2.728 tb       
raid-0         
0   L14                                  optimal         2.728 tb       
raid-0         
0   L15                                  optimal         2.728 tb       
raid-0         
0   L2                                   optimal         2.728 tb       
raid-0         
0   L3                                   optimal         2.728 tb       
raid-0         
0   L4                                   optimal         2.728 tb       
raid-0         
0   L5                                   optimal         2.728 tb       
raid-0         
0   L6                                   optimal         2.728 tb       
raid-0         
0   L7                                   optimal         2.728 tb       
raid-0         
0   L8                                   optimal         2.728 tb       
raid-0         
0   L9                                   optimal         2.728 tb       
raid-0         

---- Controller 0 Physical Drives
------------------------------------------------------------------------------------------------------
Enclosure: 245 (a0 => unavailable)
C#  Virtual Member       Slot   Model                    
Size            Serial          Media Err  Other Err 
Status               F-State  SAS Address      
0                        0      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320010
0                        1      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320011
0                        10     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320022
0                        11     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320023
0                        12     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320018
0                        13     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320019
0                        14     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001a
0                        15     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001b
0                        2      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320012
0                        3      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320013
0                        4      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000c
0                        5      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000d
0                        6      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000e
0                        7      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000f
0                        8      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320020
0                        9      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320021
  pool: storage
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Sep 17 00:20:00 2015
        14.0T scanned out of 27.1T at 1/s, (scan is slow, no estimated time)
        935G resilvered, 51.74% done
config:

    NAME        STATE     READ WRITE CKSUM
    storage     ONLINE       0     0     0
      raidz2-0  ONLINE       0     0     0
        mfid10  ONLINE       0     0     0
        mfid8   ONLINE       0     0     0
        mfid12  ONLINE       0     0     0
        mfid0   ONLINE       0     0     0
        mfid14  ONLINE       0     0     0
        mfid15  ONLINE       0     0     0
        mfid1   ONLINE       0     0     0
        mfid7   ONLINE       0     0     0
        mfid2   ONLINE       0     0     0
        mfid9   ONLINE       0     0     0
        mfid3   ONLINE       0     0     0
        mfid4   ONLINE       0     0     0
        mfid5   ONLINE       0     0     0
        mfid6   ONLINE       0     0     0
        mfid11  ONLINE       0     0     0
    spares
      mfid13    AVAIL   

errors: No known data errors
root at colossus:~ # praid ; zpool status -vx storage

MegaCli Tools Used to Gather Raid Info

==== Controllers
=======================================================================================================================
C#  Name                 CacheSize       FirmwareVer     BIOSver        
BBU            
0   lsi megaraid sas 9260-16i 512mb           2.130.403-3066  3.30.02.0
4.16.08.00 0x06060900 Missing        

---- BBUs
------------------------------------------------------------------------------------------------------------------------------
C#  Battery Type         Initialized     Voltage         Temperature    
Charge State    Alerts         
0                       
-                                                                            
 

---- Virtual Disks
---------------------------------------------------------------------------------------------------------------------
C#  Logical Volumes      Physical Disks  Degraded        Offline        
Critical        Failed         
0   16                   16              0               0              
0               0              

---- Virtual Disks Info
----------------------------------------------------------------------------------------------------------------
C#  ID                   Name            State           Size           
Raid Level     
0   L0                   mfid0           optimal         2.728 tb       
raid-0         
0   L1                                   optimal         2.728 tb       
raid-0         
0   L10                                  optimal         2.728 tb       
raid-0         
0   L11                                  optimal         2.728 tb       
raid-0         
0   L12                                  optimal         2.728 tb       
raid-0         
0   L13                                  optimal         2.728 tb       
raid-0         
0   L14                                  optimal         2.728 tb       
raid-0         
0   L15                                  optimal         2.728 tb       
raid-0         
0   L2                                   optimal         2.728 tb       
raid-0         
0   L3                                   optimal         2.728 tb       
raid-0         
0   L4                                   optimal         2.728 tb       
raid-0         
0   L5                                   optimal         2.728 tb       
raid-0         
0   L6                                   optimal         2.728 tb       
raid-0         
0   L7                                   optimal         2.728 tb       
raid-0         
0   L8                                   optimal         2.728 tb       
raid-0         
0   L9                                   optimal         2.728 tb       
raid-0         

---- Controller 0 Physical Drives
------------------------------------------------------------------------------------------------------
Enclosure: 245 (a0 => unavailable)
C#  Virtual Member       Slot   Model                    
Size            Serial          Media Err  Other Err 
Status               F-State  SAS Address      
0                        0      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320010
0                        1      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320011
0                        10     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320022
0                        11     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320023
0                        12     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320018
0                        13     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320019
0                        14     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001a
0                        15     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001b
0                        2      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320012
0                        3      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320013
0                        4      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000c
0                        5      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000d
0                        6      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000e
0                        7      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000f
0                        8      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320020
0                        9      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320021
  pool: storage
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Sep 17 00:20:00 2015
        14.1T scanned out of 27.1T at 1/s, (scan is slow, no estimated time)
        935G resilvered, 51.97% done
config:

    NAME        STATE     READ WRITE CKSUM
    storage     ONLINE       0     0     0
      raidz2-0  ONLINE       0     0     0
        mfid10  ONLINE       0     0     0
        mfid8   ONLINE       0     0     0
        mfid12  ONLINE       0     0     0
        mfid0   ONLINE       0     0     0
        mfid14  ONLINE       0     0     0
        mfid15  ONLINE       0     0     0
        mfid1   ONLINE       0     0     0
        mfid7   ONLINE       0     0     0
        mfid2   ONLINE       0     0     0
        mfid9   ONLINE       0     0     0
        mfid3   ONLINE       0     0     0
        mfid4   ONLINE       0     0     0
        mfid5   ONLINE       0     0     0
        mfid6   ONLINE       0     0     0
        mfid11  ONLINE       0     0     0
    spares
      mfid13    AVAIL   

errors: No known data errors
root at colossus:~ #


... so online (no errors) and still resilvering....!!!

Michelle


More information about the freebsd-ports mailing list