ZFS pool faulted (corrupt metadata) but the disk data appears ok...
Michelle Sullivan
michelle at sorbs.net
Thu Feb 5 22:32:19 UTC 2015
Michelle Sullivan wrote:
> Any clues on this?
>
> root at colossus:~ # zpool import
> pool: storage
> id: 10618504954404185222
> state: FAULTED
> status: The pool metadata is corrupted.
> action: The pool cannot be imported due to damaged devices or data.
> The pool may be active on another system, but can be imported using
> the '-f' flag.
> see: http://illumos.org/msg/ZFS-8000-72
> config:
>
> storage FAULTED corrupted data
> raidz2-0 ONLINE
> mfid0 ONLINE
> mfid15 ONLINE
> mfid1 ONLINE
> mfid2 ONLINE
> mfid3 ONLINE
> mfid4 ONLINE
> mfid5 ONLINE
> replacing-7 ONLINE
> mfid13 ONLINE
> mfid14 ONLINE
> mfid6 ONLINE
> mfid7 ONLINE
> mfid8 ONLINE
> mfid9 ONLINE
> mfid10 ONLINE
> mfid11 ONLINE
> mfid12 ONLINE
> root at colossus:~ # zpool import -Ff storage
> cannot import 'storage': I/O error
> Destroy and re-create the pool from
> a backup source.
> root at colossus:~ # zdb -l /dev/mfid0
> --------------------------------------------
> LABEL 0
> --------------------------------------------
> version: 5000
> name: 'storage'
> state: 0
> txg: 1928241
> pool_guid: 10618504954404185222
> hostid: 4203774842
> hostname: 'colossus'
> top_guid: 12489400212295803034
> guid: 3998695725653225547
> vdev_children: 1
> vdev_tree:
> type: 'raidz'
> id: 0
> guid: 12489400212295803034
> nparity: 2
> metaslab_array: 34
> metaslab_shift: 38
> ashift: 9
> asize: 45000449064960
> is_log: 0
> create_txg: 4
> children[0]:
> type: 'disk'
> id: 0
> guid: 3998695725653225547
> path: '/dev/mfid0'
> phys_path: '/dev/mfid0'
> whole_disk: 1
> DTL: 168
> create_txg: 4
> children[1]:
> type: 'disk'
> id: 1
> guid: 10795471632546545577
> path: '/dev/mfid1'
> phys_path: '/dev/mfid1'
> whole_disk: 1
> DTL: 167
> create_txg: 4
> children[2]:
> type: 'disk'
> id: 2
> guid: 15820272272734706674
> path: '/dev/mfid2'
> phys_path: '/dev/mfid2'
> whole_disk: 1
> DTL: 166
> create_txg: 4
> children[3]:
> type: 'disk'
> id: 3
> guid: 3928579496187019848
> path: '/dev/mfid3'
> phys_path: '/dev/mfid3'
> whole_disk: 1
> DTL: 165
> create_txg: 4
> children[4]:
> type: 'disk'
> id: 4
> guid: 7125052278051590304
> path: '/dev/mfid4'
> phys_path: '/dev/mfid4'
> whole_disk: 1
> DTL: 164
> create_txg: 4
> children[5]:
> type: 'disk'
> id: 5
> guid: 14370198745088794709
> path: '/dev/mfid5'
> phys_path: '/dev/mfid5'
> whole_disk: 1
> DTL: 163
> create_txg: 4
> children[6]:
> type: 'disk'
> id: 6
> guid: 1843597351388951655
> path: '/dev/mfid6'
> phys_path: '/dev/mfid6'
> whole_disk: 1
> DTL: 162
> create_txg: 4
> children[7]:
> type: 'replacing'
> id: 7
> guid: 2914889727426054645
> whole_disk: 0
> create_txg: 4
> children[0]:
> type: 'disk'
> id: 0
> guid: 10956220251832269421
> path: '/dev/mfid15'
> phys_path: '/dev/mfid15'
> whole_disk: 1
> DTL: 179
> create_txg: 4
> children[1]:
> type: 'disk'
> id: 1
> guid: 2463756237300743131
> path: '/dev/mfid13'
> phys_path: '/dev/mfid13'
> whole_disk: 1
> DTL: 181
> create_txg: 4
> resilvering: 1
> children[8]:
> type: 'disk'
> id: 8
> guid: 8864096842672670007
> path: '/dev/mfid7'
> phys_path: '/dev/mfid7'
> whole_disk: 1
> DTL: 160
> create_txg: 4
> children[9]:
> type: 'disk'
> id: 9
> guid: 4650681673751655245
> path: '/dev/mfid8'
> phys_path: '/dev/mfid8'
> whole_disk: 1
> DTL: 159
> create_txg: 4
> children[10]:
> type: 'disk'
> id: 10
> guid: 8432109430432996813
> path: '/dev/mfid9'
> phys_path: '/dev/mfid9'
> whole_disk: 1
> DTL: 158
> create_txg: 4
> children[11]:
> type: 'disk'
> id: 11
> guid: 414941847968750824
> path: '/dev/mfid10'
> phys_path: '/dev/mfid10'
> whole_disk: 1
> DTL: 157
> create_txg: 4
> children[12]:
> type: 'disk'
> id: 12
> guid: 7335375930620195352
> path: '/dev/mfid11'
> phys_path: '/dev/mfid11'
> whole_disk: 1
> DTL: 156
> create_txg: 4
> children[13]:
> type: 'disk'
> id: 13
> guid: 5100737174610362
> path: '/dev/mfid12'
> phys_path: '/dev/mfid12'
> whole_disk: 1
> DTL: 155
> create_txg: 4
> children[14]:
> type: 'disk'
> id: 14
> guid: 15695558693726858796
> path: '/dev/mfid14'
> phys_path: '/dev/mfid14'
> whole_disk: 1
> DTL: 174
> create_txg: 4
> features_for_read:
> --------------------------------------------
> LABEL 1
> --------------------------------------------
> version: 5000
> name: 'storage'
> state: 0
> txg: 1928241
> pool_guid: 10618504954404185222
> hostid: 4203774842
> hostname: 'colossus'
> top_guid: 12489400212295803034
> guid: 3998695725653225547
> vdev_children: 1
> vdev_tree:
> type: 'raidz'
> id: 0
> guid: 12489400212295803034
> nparity: 2
> metaslab_array: 34
> metaslab_shift: 38
> ashift: 9
> asize: 45000449064960
> is_log: 0
> create_txg: 4
> children[0]:
> type: 'disk'
> id: 0
> guid: 3998695725653225547
> path: '/dev/mfid0'
> phys_path: '/dev/mfid0'
> whole_disk: 1
> DTL: 168
> create_txg: 4
> children[1]:
> type: 'disk'
> id: 1
> guid: 10795471632546545577
> path: '/dev/mfid1'
> phys_path: '/dev/mfid1'
> whole_disk: 1
> DTL: 167
> create_txg: 4
> children[2]:
> type: 'disk'
> id: 2
> guid: 15820272272734706674
> path: '/dev/mfid2'
> phys_path: '/dev/mfid2'
> whole_disk: 1
> DTL: 166
> create_txg: 4
> children[3]:
> type: 'disk'
> id: 3
> guid: 3928579496187019848
> path: '/dev/mfid3'
> phys_path: '/dev/mfid3'
> whole_disk: 1
> DTL: 165
> create_txg: 4
> children[4]:
> type: 'disk'
> id: 4
> guid: 7125052278051590304
> path: '/dev/mfid4'
> phys_path: '/dev/mfid4'
> whole_disk: 1
> DTL: 164
> create_txg: 4
> children[5]:
> type: 'disk'
> id: 5
> guid: 14370198745088794709
> path: '/dev/mfid5'
> phys_path: '/dev/mfid5'
> whole_disk: 1
> DTL: 163
> create_txg: 4
> children[6]:
> type: 'disk'
> id: 6
> guid: 1843597351388951655
> path: '/dev/mfid6'
> phys_path: '/dev/mfid6'
> whole_disk: 1
> DTL: 162
> create_txg: 4
> children[7]:
> type: 'replacing'
> id: 7
> guid: 2914889727426054645
> whole_disk: 0
> create_txg: 4
> children[0]:
> type: 'disk'
> id: 0
> guid: 10956220251832269421
> path: '/dev/mfid15'
> phys_path: '/dev/mfid15'
> whole_disk: 1
> DTL: 179
> create_txg: 4
> children[1]:
> type: 'disk'
> id: 1
> guid: 2463756237300743131
> path: '/dev/mfid13'
> phys_path: '/dev/mfid13'
> whole_disk: 1
> DTL: 181
> create_txg: 4
> resilvering: 1
> children[8]:
> type: 'disk'
> id: 8
> guid: 8864096842672670007
> path: '/dev/mfid7'
> phys_path: '/dev/mfid7'
> whole_disk: 1
> DTL: 160
> create_txg: 4
> children[9]:
> type: 'disk'
> id: 9
> guid: 4650681673751655245
> path: '/dev/mfid8'
> phys_path: '/dev/mfid8'
> whole_disk: 1
> DTL: 159
> create_txg: 4
> children[10]:
> type: 'disk'
> id: 10
> guid: 8432109430432996813
> path: '/dev/mfid9'
> phys_path: '/dev/mfid9'
> whole_disk: 1
> DTL: 158
> create_txg: 4
> children[11]:
> type: 'disk'
> id: 11
> guid: 414941847968750824
> path: '/dev/mfid10'
> phys_path: '/dev/mfid10'
> whole_disk: 1
> DTL: 157
> create_txg: 4
> children[12]:
> type: 'disk'
> id: 12
> guid: 7335375930620195352
> path: '/dev/mfid11'
> phys_path: '/dev/mfid11'
> whole_disk: 1
> DTL: 156
> create_txg: 4
> children[13]:
> type: 'disk'
> id: 13
> guid: 5100737174610362
> path: '/dev/mfid12'
> phys_path: '/dev/mfid12'
> whole_disk: 1
> DTL: 155
> create_txg: 4
> children[14]:
> type: 'disk'
> id: 14
> guid: 15695558693726858796
> path: '/dev/mfid14'
> phys_path: '/dev/mfid14'
> whole_disk: 1
> DTL: 174
> create_txg: 4
> features_for_read:
> --------------------------------------------
> LABEL 2
> --------------------------------------------
> version: 5000
> name: 'storage'
> state: 0
> txg: 1928241
> pool_guid: 10618504954404185222
> hostid: 4203774842
> hostname: 'colossus'
> top_guid: 12489400212295803034
> guid: 3998695725653225547
> vdev_children: 1
> vdev_tree:
> type: 'raidz'
> id: 0
> guid: 12489400212295803034
> nparity: 2
> metaslab_array: 34
> metaslab_shift: 38
> ashift: 9
> asize: 45000449064960
> is_log: 0
> create_txg: 4
> children[0]:
> type: 'disk'
> id: 0
> guid: 3998695725653225547
> path: '/dev/mfid0'
> phys_path: '/dev/mfid0'
> whole_disk: 1
> DTL: 168
> create_txg: 4
> children[1]:
> type: 'disk'
> id: 1
> guid: 10795471632546545577
> path: '/dev/mfid1'
> phys_path: '/dev/mfid1'
> whole_disk: 1
> DTL: 167
> create_txg: 4
> children[2]:
> type: 'disk'
> id: 2
> guid: 15820272272734706674
> path: '/dev/mfid2'
> phys_path: '/dev/mfid2'
> whole_disk: 1
> DTL: 166
> create_txg: 4
> children[3]:
> type: 'disk'
> id: 3
> guid: 3928579496187019848
> path: '/dev/mfid3'
> phys_path: '/dev/mfid3'
> whole_disk: 1
> DTL: 165
> create_txg: 4
> children[4]:
> type: 'disk'
> id: 4
> guid: 7125052278051590304
> path: '/dev/mfid4'
> phys_path: '/dev/mfid4'
> whole_disk: 1
> DTL: 164
> create_txg: 4
> children[5]:
> type: 'disk'
> id: 5
> guid: 14370198745088794709
> path: '/dev/mfid5'
> phys_path: '/dev/mfid5'
> whole_disk: 1
> DTL: 163
> create_txg: 4
> children[6]:
> type: 'disk'
> id: 6
> guid: 1843597351388951655
> path: '/dev/mfid6'
> phys_path: '/dev/mfid6'
> whole_disk: 1
> DTL: 162
> create_txg: 4
> children[7]:
> type: 'replacing'
> id: 7
> guid: 2914889727426054645
> whole_disk: 0
> create_txg: 4
> children[0]:
> type: 'disk'
> id: 0
> guid: 10956220251832269421
> path: '/dev/mfid15'
> phys_path: '/dev/mfid15'
> whole_disk: 1
> DTL: 179
> create_txg: 4
> children[1]:
> type: 'disk'
> id: 1
> guid: 2463756237300743131
> path: '/dev/mfid13'
> phys_path: '/dev/mfid13'
> whole_disk: 1
> DTL: 181
> create_txg: 4
> resilvering: 1
> children[8]:
> type: 'disk'
> id: 8
> guid: 8864096842672670007
> path: '/dev/mfid7'
> phys_path: '/dev/mfid7'
> whole_disk: 1
> DTL: 160
> create_txg: 4
> children[9]:
> type: 'disk'
> id: 9
> guid: 4650681673751655245
> path: '/dev/mfid8'
> phys_path: '/dev/mfid8'
> whole_disk: 1
> DTL: 159
> create_txg: 4
> children[10]:
> type: 'disk'
> id: 10
> guid: 8432109430432996813
> path: '/dev/mfid9'
> phys_path: '/dev/mfid9'
> whole_disk: 1
> DTL: 158
> create_txg: 4
> children[11]:
> type: 'disk'
> id: 11
> guid: 414941847968750824
> path: '/dev/mfid10'
> phys_path: '/dev/mfid10'
> whole_disk: 1
> DTL: 157
> create_txg: 4
> children[12]:
> type: 'disk'
> id: 12
> guid: 7335375930620195352
> path: '/dev/mfid11'
> phys_path: '/dev/mfid11'
> whole_disk: 1
> DTL: 156
> create_txg: 4
> children[13]:
> type: 'disk'
> id: 13
> guid: 5100737174610362
> path: '/dev/mfid12'
> phys_path: '/dev/mfid12'
> whole_disk: 1
> DTL: 155
> create_txg: 4
> children[14]:
> type: 'disk'
> id: 14
> guid: 15695558693726858796
> path: '/dev/mfid14'
> phys_path: '/dev/mfid14'
> whole_disk: 1
> DTL: 174
> create_txg: 4
> features_for_read:
> --------------------------------------------
> LABEL 3
> --------------------------------------------
> version: 5000
> name: 'storage'
> state: 0
> txg: 1928241
> pool_guid: 10618504954404185222
> hostid: 4203774842
> hostname: 'colossus'
> top_guid: 12489400212295803034
> guid: 3998695725653225547
> vdev_children: 1
> vdev_tree:
> type: 'raidz'
> id: 0
> guid: 12489400212295803034
> nparity: 2
> metaslab_array: 34
> metaslab_shift: 38
> ashift: 9
> asize: 45000449064960
> is_log: 0
> create_txg: 4
> children[0]:
> type: 'disk'
> id: 0
> guid: 3998695725653225547
> path: '/dev/mfid0'
> phys_path: '/dev/mfid0'
> whole_disk: 1
> DTL: 168
> create_txg: 4
> children[1]:
> type: 'disk'
> id: 1
> guid: 10795471632546545577
> path: '/dev/mfid1'
> phys_path: '/dev/mfid1'
> whole_disk: 1
> DTL: 167
> create_txg: 4
> children[2]:
> type: 'disk'
> id: 2
> guid: 15820272272734706674
> path: '/dev/mfid2'
> phys_path: '/dev/mfid2'
> whole_disk: 1
> DTL: 166
> create_txg: 4
> children[3]:
> type: 'disk'
> id: 3
> guid: 3928579496187019848
> path: '/dev/mfid3'
> phys_path: '/dev/mfid3'
> whole_disk: 1
> DTL: 165
> create_txg: 4
> children[4]:
> type: 'disk'
> id: 4
> guid: 7125052278051590304
> path: '/dev/mfid4'
> phys_path: '/dev/mfid4'
> whole_disk: 1
> DTL: 164
> create_txg: 4
> children[5]:
> type: 'disk'
> id: 5
> guid: 14370198745088794709
> path: '/dev/mfid5'
> phys_path: '/dev/mfid5'
> whole_disk: 1
> DTL: 163
> create_txg: 4
> children[6]:
> type: 'disk'
> id: 6
> guid: 1843597351388951655
> path: '/dev/mfid6'
> phys_path: '/dev/mfid6'
> whole_disk: 1
> DTL: 162
> create_txg: 4
> children[7]:
> type: 'replacing'
> id: 7
> guid: 2914889727426054645
> whole_disk: 0
> create_txg: 4
> children[0]:
> type: 'disk'
> id: 0
> guid: 10956220251832269421
> path: '/dev/mfid15'
> phys_path: '/dev/mfid15'
> whole_disk: 1
> DTL: 179
> create_txg: 4
> children[1]:
> type: 'disk'
> id: 1
> guid: 2463756237300743131
> path: '/dev/mfid13'
> phys_path: '/dev/mfid13'
> whole_disk: 1
> DTL: 181
> create_txg: 4
> resilvering: 1
> children[8]:
> type: 'disk'
> id: 8
> guid: 8864096842672670007
> path: '/dev/mfid7'
> phys_path: '/dev/mfid7'
> whole_disk: 1
> DTL: 160
> create_txg: 4
> children[9]:
> type: 'disk'
> id: 9
> guid: 4650681673751655245
> path: '/dev/mfid8'
> phys_path: '/dev/mfid8'
> whole_disk: 1
> DTL: 159
> create_txg: 4
> children[10]:
> type: 'disk'
> id: 10
> guid: 8432109430432996813
> path: '/dev/mfid9'
> phys_path: '/dev/mfid9'
> whole_disk: 1
> DTL: 158
> create_txg: 4
> children[11]:
> type: 'disk'
> id: 11
> guid: 414941847968750824
> path: '/dev/mfid10'
> phys_path: '/dev/mfid10'
> whole_disk: 1
> DTL: 157
> create_txg: 4
> children[12]:
> type: 'disk'
> id: 12
> guid: 7335375930620195352
> path: '/dev/mfid11'
> phys_path: '/dev/mfid11'
> whole_disk: 1
> DTL: 156
> create_txg: 4
> children[13]:
> type: 'disk'
> id: 13
> guid: 5100737174610362
> path: '/dev/mfid12'
> phys_path: '/dev/mfid12'
> whole_disk: 1
> DTL: 155
> create_txg: 4
> children[14]:
> type: 'disk'
> id: 14
> guid: 15695558693726858796
> path: '/dev/mfid14'
> phys_path: '/dev/mfid14'
> whole_disk: 1
> DTL: 174
> create_txg: 4
> features_for_read:
>
> LSI9260-16i raid controller all drives in single disk RAID0, system
> wasn't busy (=in use for writing, however was resilvering a replacement
> drive after the HSP had taken over due to failed drive) and it got
> rebooted... faulted on reboot.
>
> Recoverable?
>
> Regards,
>
>
FYI before the first reboot:
root at colossus:~ # zpool status -x
pool: storage
state: FAULTED
status: The pool metadata is corrupted and the pool cannot be opened.
action: Destroy and re-create the pool from
a backup source.
see: http://illumos.org/msg/ZFS-8000-72
scan: none requested
config:
NAME STATE READ WRITE CKSUM
storage FAULTED 0 0 1
raidz2-0 ONLINE 0 0 7
mfid0 ONLINE 0 0 1
mfid1 ONLINE 0 0 0
mfid2 ONLINE 0 0 0
mfid3 ONLINE 0 0 0
mfid4 ONLINE 0 0 0
mfid5 ONLINE 0 0 0
mfid6 ONLINE 0 0 0
replacing-7 ONLINE 0 0 0
mfid14 ONLINE 0 0 0
mfid15 ONLINE 0 0 0
mfid7 ONLINE 0 0 0
mfid8 ONLINE 0 0 0
mfid9 ONLINE 0 0 0
mfid10 ONLINE 0 0 0
mfid11 ONLINE 0 0 0
mfid12 ONLINE 0 0 1
mfid13 ONLINE 0 0 0
root at colossus:~ # zpool clear -nF storage
internal error: out of memory
root at colossus:~ #
"zpool import -Fn storage" reports no errors.
Really don't care if a file or three is corrupted, just want the pool
back as don't have a backup since Dec 3, 2014.
--
Michelle Sullivan
http://www.mhix.org/
More information about the freebsd-fs
mailing list