ZFS pool faulted (corrupt metadata) but the disk data appears ok...

Michelle Sullivan michelle at sorbs.net
Thu Feb 5 22:09:04 UTC 2015


Any clues on this?

root at colossus:~ # zpool import
   pool: storage
     id: 10618504954404185222
  state: FAULTED
 status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
    The pool may be active on another system, but can be imported using
    the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-72
 config:

    storage          FAULTED  corrupted data
      raidz2-0       ONLINE
        mfid0        ONLINE
        mfid15       ONLINE
        mfid1        ONLINE
        mfid2        ONLINE
        mfid3        ONLINE
        mfid4        ONLINE
        mfid5        ONLINE
        replacing-7  ONLINE
          mfid13     ONLINE
          mfid14     ONLINE
        mfid6        ONLINE
        mfid7        ONLINE
        mfid8        ONLINE
        mfid9        ONLINE
        mfid10       ONLINE
        mfid11       ONLINE
        mfid12       ONLINE
root at colossus:~ # zpool import -Ff storage
cannot import 'storage': I/O error
    Destroy and re-create the pool from
    a backup source.
root at colossus:~ # zdb -l /dev/mfid0
--------------------------------------------
LABEL 0
--------------------------------------------
    version: 5000
    name: 'storage'
    state: 0
    txg: 1928241
    pool_guid: 10618504954404185222
    hostid: 4203774842
    hostname: 'colossus'
    top_guid: 12489400212295803034
    guid: 3998695725653225547
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 12489400212295803034
        nparity: 2
        metaslab_array: 34
        metaslab_shift: 38
        ashift: 9
        asize: 45000449064960
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 3998695725653225547
            path: '/dev/mfid0'
            phys_path: '/dev/mfid0'
            whole_disk: 1
            DTL: 168
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 10795471632546545577
            path: '/dev/mfid1'
            phys_path: '/dev/mfid1'
            whole_disk: 1
            DTL: 167
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 15820272272734706674
            path: '/dev/mfid2'
            phys_path: '/dev/mfid2'
            whole_disk: 1
            DTL: 166
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 3928579496187019848
            path: '/dev/mfid3'
            phys_path: '/dev/mfid3'
            whole_disk: 1
            DTL: 165
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 7125052278051590304
            path: '/dev/mfid4'
            phys_path: '/dev/mfid4'
            whole_disk: 1
            DTL: 164
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 14370198745088794709
            path: '/dev/mfid5'
            phys_path: '/dev/mfid5'
            whole_disk: 1
            DTL: 163
            create_txg: 4
        children[6]:
            type: 'disk'
            id: 6
            guid: 1843597351388951655
            path: '/dev/mfid6'
            phys_path: '/dev/mfid6'
            whole_disk: 1
            DTL: 162
            create_txg: 4
        children[7]:
            type: 'replacing'
            id: 7
            guid: 2914889727426054645
            whole_disk: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 10956220251832269421
                path: '/dev/mfid15'
                phys_path: '/dev/mfid15'
                whole_disk: 1
                DTL: 179
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 2463756237300743131
                path: '/dev/mfid13'
                phys_path: '/dev/mfid13'
                whole_disk: 1
                DTL: 181
                create_txg: 4
                resilvering: 1
        children[8]:
            type: 'disk'
            id: 8
            guid: 8864096842672670007
            path: '/dev/mfid7'
            phys_path: '/dev/mfid7'
            whole_disk: 1
            DTL: 160
            create_txg: 4
        children[9]:
            type: 'disk'
            id: 9
            guid: 4650681673751655245
            path: '/dev/mfid8'
            phys_path: '/dev/mfid8'
            whole_disk: 1
            DTL: 159
            create_txg: 4
        children[10]:
            type: 'disk'
            id: 10
            guid: 8432109430432996813
            path: '/dev/mfid9'
            phys_path: '/dev/mfid9'
            whole_disk: 1
            DTL: 158
            create_txg: 4
        children[11]:
            type: 'disk'
            id: 11
            guid: 414941847968750824
            path: '/dev/mfid10'
            phys_path: '/dev/mfid10'
            whole_disk: 1
            DTL: 157
            create_txg: 4
        children[12]:
            type: 'disk'
            id: 12
            guid: 7335375930620195352
            path: '/dev/mfid11'
            phys_path: '/dev/mfid11'
            whole_disk: 1
            DTL: 156
            create_txg: 4
        children[13]:
            type: 'disk'
            id: 13
            guid: 5100737174610362
            path: '/dev/mfid12'
            phys_path: '/dev/mfid12'
            whole_disk: 1
            DTL: 155
            create_txg: 4
        children[14]:
            type: 'disk'
            id: 14
            guid: 15695558693726858796
            path: '/dev/mfid14'
            phys_path: '/dev/mfid14'
            whole_disk: 1
            DTL: 174
            create_txg: 4
    features_for_read:
--------------------------------------------
LABEL 1
--------------------------------------------
    version: 5000
    name: 'storage'
    state: 0
    txg: 1928241
    pool_guid: 10618504954404185222
    hostid: 4203774842
    hostname: 'colossus'
    top_guid: 12489400212295803034
    guid: 3998695725653225547
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 12489400212295803034
        nparity: 2
        metaslab_array: 34
        metaslab_shift: 38
        ashift: 9
        asize: 45000449064960
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 3998695725653225547
            path: '/dev/mfid0'
            phys_path: '/dev/mfid0'
            whole_disk: 1
            DTL: 168
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 10795471632546545577
            path: '/dev/mfid1'
            phys_path: '/dev/mfid1'
            whole_disk: 1
            DTL: 167
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 15820272272734706674
            path: '/dev/mfid2'
            phys_path: '/dev/mfid2'
            whole_disk: 1
            DTL: 166
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 3928579496187019848
            path: '/dev/mfid3'
            phys_path: '/dev/mfid3'
            whole_disk: 1
            DTL: 165
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 7125052278051590304
            path: '/dev/mfid4'
            phys_path: '/dev/mfid4'
            whole_disk: 1
            DTL: 164
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 14370198745088794709
            path: '/dev/mfid5'
            phys_path: '/dev/mfid5'
            whole_disk: 1
            DTL: 163
            create_txg: 4
        children[6]:
            type: 'disk'
            id: 6
            guid: 1843597351388951655
            path: '/dev/mfid6'
            phys_path: '/dev/mfid6'
            whole_disk: 1
            DTL: 162
            create_txg: 4
        children[7]:
            type: 'replacing'
            id: 7
            guid: 2914889727426054645
            whole_disk: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 10956220251832269421
                path: '/dev/mfid15'
                phys_path: '/dev/mfid15'
                whole_disk: 1
                DTL: 179
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 2463756237300743131
                path: '/dev/mfid13'
                phys_path: '/dev/mfid13'
                whole_disk: 1
                DTL: 181
                create_txg: 4
                resilvering: 1
        children[8]:
            type: 'disk'
            id: 8
            guid: 8864096842672670007
            path: '/dev/mfid7'
            phys_path: '/dev/mfid7'
            whole_disk: 1
            DTL: 160
            create_txg: 4
        children[9]:
            type: 'disk'
            id: 9
            guid: 4650681673751655245
            path: '/dev/mfid8'
            phys_path: '/dev/mfid8'
            whole_disk: 1
            DTL: 159
            create_txg: 4
        children[10]:
            type: 'disk'
            id: 10
            guid: 8432109430432996813
            path: '/dev/mfid9'
            phys_path: '/dev/mfid9'
            whole_disk: 1
            DTL: 158
            create_txg: 4
        children[11]:
            type: 'disk'
            id: 11
            guid: 414941847968750824
            path: '/dev/mfid10'
            phys_path: '/dev/mfid10'
            whole_disk: 1
            DTL: 157
            create_txg: 4
        children[12]:
            type: 'disk'
            id: 12
            guid: 7335375930620195352
            path: '/dev/mfid11'
            phys_path: '/dev/mfid11'
            whole_disk: 1
            DTL: 156
            create_txg: 4
        children[13]:
            type: 'disk'
            id: 13
            guid: 5100737174610362
            path: '/dev/mfid12'
            phys_path: '/dev/mfid12'
            whole_disk: 1
            DTL: 155
            create_txg: 4
        children[14]:
            type: 'disk'
            id: 14
            guid: 15695558693726858796
            path: '/dev/mfid14'
            phys_path: '/dev/mfid14'
            whole_disk: 1
            DTL: 174
            create_txg: 4
    features_for_read:
--------------------------------------------
LABEL 2
--------------------------------------------
    version: 5000
    name: 'storage'
    state: 0
    txg: 1928241
    pool_guid: 10618504954404185222
    hostid: 4203774842
    hostname: 'colossus'
    top_guid: 12489400212295803034
    guid: 3998695725653225547
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 12489400212295803034
        nparity: 2
        metaslab_array: 34
        metaslab_shift: 38
        ashift: 9
        asize: 45000449064960
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 3998695725653225547
            path: '/dev/mfid0'
            phys_path: '/dev/mfid0'
            whole_disk: 1
            DTL: 168
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 10795471632546545577
            path: '/dev/mfid1'
            phys_path: '/dev/mfid1'
            whole_disk: 1
            DTL: 167
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 15820272272734706674
            path: '/dev/mfid2'
            phys_path: '/dev/mfid2'
            whole_disk: 1
            DTL: 166
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 3928579496187019848
            path: '/dev/mfid3'
            phys_path: '/dev/mfid3'
            whole_disk: 1
            DTL: 165
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 7125052278051590304
            path: '/dev/mfid4'
            phys_path: '/dev/mfid4'
            whole_disk: 1
            DTL: 164
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 14370198745088794709
            path: '/dev/mfid5'
            phys_path: '/dev/mfid5'
            whole_disk: 1
            DTL: 163
            create_txg: 4
        children[6]:
            type: 'disk'
            id: 6
            guid: 1843597351388951655
            path: '/dev/mfid6'
            phys_path: '/dev/mfid6'
            whole_disk: 1
            DTL: 162
            create_txg: 4
        children[7]:
            type: 'replacing'
            id: 7
            guid: 2914889727426054645
            whole_disk: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 10956220251832269421
                path: '/dev/mfid15'
                phys_path: '/dev/mfid15'
                whole_disk: 1
                DTL: 179
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 2463756237300743131
                path: '/dev/mfid13'
                phys_path: '/dev/mfid13'
                whole_disk: 1
                DTL: 181
                create_txg: 4
                resilvering: 1
        children[8]:
            type: 'disk'
            id: 8
            guid: 8864096842672670007
            path: '/dev/mfid7'
            phys_path: '/dev/mfid7'
            whole_disk: 1
            DTL: 160
            create_txg: 4
        children[9]:
            type: 'disk'
            id: 9
            guid: 4650681673751655245
            path: '/dev/mfid8'
            phys_path: '/dev/mfid8'
            whole_disk: 1
            DTL: 159
            create_txg: 4
        children[10]:
            type: 'disk'
            id: 10
            guid: 8432109430432996813
            path: '/dev/mfid9'
            phys_path: '/dev/mfid9'
            whole_disk: 1
            DTL: 158
            create_txg: 4
        children[11]:
            type: 'disk'
            id: 11
            guid: 414941847968750824
            path: '/dev/mfid10'
            phys_path: '/dev/mfid10'
            whole_disk: 1
            DTL: 157
            create_txg: 4
        children[12]:
            type: 'disk'
            id: 12
            guid: 7335375930620195352
            path: '/dev/mfid11'
            phys_path: '/dev/mfid11'
            whole_disk: 1
            DTL: 156
            create_txg: 4
        children[13]:
            type: 'disk'
            id: 13
            guid: 5100737174610362
            path: '/dev/mfid12'
            phys_path: '/dev/mfid12'
            whole_disk: 1
            DTL: 155
            create_txg: 4
        children[14]:
            type: 'disk'
            id: 14
            guid: 15695558693726858796
            path: '/dev/mfid14'
            phys_path: '/dev/mfid14'
            whole_disk: 1
            DTL: 174
            create_txg: 4
    features_for_read:
--------------------------------------------
LABEL 3
--------------------------------------------
    version: 5000
    name: 'storage'
    state: 0
    txg: 1928241
    pool_guid: 10618504954404185222
    hostid: 4203774842
    hostname: 'colossus'
    top_guid: 12489400212295803034
    guid: 3998695725653225547
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 12489400212295803034
        nparity: 2
        metaslab_array: 34
        metaslab_shift: 38
        ashift: 9
        asize: 45000449064960
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 3998695725653225547
            path: '/dev/mfid0'
            phys_path: '/dev/mfid0'
            whole_disk: 1
            DTL: 168
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 10795471632546545577
            path: '/dev/mfid1'
            phys_path: '/dev/mfid1'
            whole_disk: 1
            DTL: 167
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 15820272272734706674
            path: '/dev/mfid2'
            phys_path: '/dev/mfid2'
            whole_disk: 1
            DTL: 166
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 3928579496187019848
            path: '/dev/mfid3'
            phys_path: '/dev/mfid3'
            whole_disk: 1
            DTL: 165
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 7125052278051590304
            path: '/dev/mfid4'
            phys_path: '/dev/mfid4'
            whole_disk: 1
            DTL: 164
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 14370198745088794709
            path: '/dev/mfid5'
            phys_path: '/dev/mfid5'
            whole_disk: 1
            DTL: 163
            create_txg: 4
        children[6]:
            type: 'disk'
            id: 6
            guid: 1843597351388951655
            path: '/dev/mfid6'
            phys_path: '/dev/mfid6'
            whole_disk: 1
            DTL: 162
            create_txg: 4
        children[7]:
            type: 'replacing'
            id: 7
            guid: 2914889727426054645
            whole_disk: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 10956220251832269421
                path: '/dev/mfid15'
                phys_path: '/dev/mfid15'
                whole_disk: 1
                DTL: 179
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 2463756237300743131
                path: '/dev/mfid13'
                phys_path: '/dev/mfid13'
                whole_disk: 1
                DTL: 181
                create_txg: 4
                resilvering: 1
        children[8]:
            type: 'disk'
            id: 8
            guid: 8864096842672670007
            path: '/dev/mfid7'
            phys_path: '/dev/mfid7'
            whole_disk: 1
            DTL: 160
            create_txg: 4
        children[9]:
            type: 'disk'
            id: 9
            guid: 4650681673751655245
            path: '/dev/mfid8'
            phys_path: '/dev/mfid8'
            whole_disk: 1
            DTL: 159
            create_txg: 4
        children[10]:
            type: 'disk'
            id: 10
            guid: 8432109430432996813
            path: '/dev/mfid9'
            phys_path: '/dev/mfid9'
            whole_disk: 1
            DTL: 158
            create_txg: 4
        children[11]:
            type: 'disk'
            id: 11
            guid: 414941847968750824
            path: '/dev/mfid10'
            phys_path: '/dev/mfid10'
            whole_disk: 1
            DTL: 157
            create_txg: 4
        children[12]:
            type: 'disk'
            id: 12
            guid: 7335375930620195352
            path: '/dev/mfid11'
            phys_path: '/dev/mfid11'
            whole_disk: 1
            DTL: 156
            create_txg: 4
        children[13]:
            type: 'disk'
            id: 13
            guid: 5100737174610362
            path: '/dev/mfid12'
            phys_path: '/dev/mfid12'
            whole_disk: 1
            DTL: 155
            create_txg: 4
        children[14]:
            type: 'disk'
            id: 14
            guid: 15695558693726858796
            path: '/dev/mfid14'
            phys_path: '/dev/mfid14'
            whole_disk: 1
            DTL: 174
            create_txg: 4
    features_for_read:

LSI9260-16i raid controller all drives in single disk RAID0, system
wasn't busy (=in use for writing, however was resilvering a replacement
drive after the HSP had taken over due to failed drive) and it got
rebooted... faulted on reboot.

Recoverable?

Regards,

-- 
Michelle Sullivan
http://www.mhix.org/



More information about the freebsd-fs mailing list