zpool import failes with internal error: Unknown Error 122

[Andriy Gapon]

On 12/07/2018 22:28, freebsdlists at bsdunix.ch wrote:
> 
> Hi,
> 
> I can't import an exported raidz1 pool based on 4 disk. An zpool export was done
> with Freebsd 11.2 and an import too. One disk (ada3 aka
> /dev/diskid/DISK-ZA18YNHS) was marked as unavailable before the export was done.
> 
> This particular disk was offlined before the export and a zpool labelclear ada3
> was done too before the export.
> 
> I'm unable to import the pool again. 3 of 4 disk are available in raidz1
> 
> "zool import storage" command:
>    pool: storage
>      id: 9514877379131531055
>   state: FAULTED
>  status: One or more devices are missing from the system.
>  action: The pool cannot be imported. Attach the missing
>     devices and try again.
>    see: http://illumos.org/msg/ZFS-8000-3C
>  config:
> 
>     storage                   FAULTED  corrupted data
>       raidz1-0                FAULTED  corrupted data
>         1800501377521064476   UNAVAIL  cannot open
>         diskid/DISK-ZA16XZW9  ONLINE
>         diskid/DISK-ZA18JE4E  ONLINE
>         diskid/DISK-ZA18Z0L1  ONLINE
> 
> Disk 1800501377521064476 is ada3 aka /dev/diskid/DISK-ZA18YNH.
> 
> I tried to import the pool with -F, -fF, -FfX ...
> 
> zool import -f storage fails too.
> internal error: Unknown Error 122
> Abort (core dumped)
> 
> Zdb showes:
> zdb -Fe storage gives:
> Configuration for import:
>         vdev_children: 1
>         version: 5000
>         pool_guid: 9514877379131531055
>         name: 'storage'
>         state: 1
>         hostid: 2232729950
>         hostname: 'my.machine.com'
>         vdev_tree:
>             type: 'root'
>             id: 0
>             guid: 9514877379131531055
>             children[0]:
>                 type: 'raidz'
>                 id: 0
>                 guid: 12743926568081883225
>                 nparity: 1
>                 metaslab_array: 40
>                 metaslab_shift: 38
>                 ashift: 12
>                 asize: 32006233653248
>                 is_log: 0
>                 create_txg: 4
>                 children[0]:
>                     type: 'disk'
>                     id: 0
>                     guid: 1800501377521064476
>                     path: '/dev/diskid/DISK-ZA18YNHS'
>                     whole_disk: 1
>                     DTL: 58
>                     create_txg: 4
>                 children[1]:
>                     type: 'disk'
>                     id: 1
>                     guid: 3017550635642283205
>                     whole_disk: 1
>                     DTL: 57
>                     create_txg: 4
>                     path: '/dev/diskid/DISK-ZA16XZW9'
>                 children[2]:
>                     type: 'disk'
>                     id: 2
>                     guid: 1070420248480515562
>                     whole_disk: 1
>                     DTL: 56
>                     create_txg: 4
>                     path: '/dev/diskid/DISK-ZA18JE4E'
>                 children[3]:
>                     type: 'disk'
>                     id: 3
>                     guid: 1151953985479393249
>                     whole_disk: 1
>                     DTL: 55
>                     create_txg: 4
>                     path: '/dev/diskid/DISK-ZA18Z0L1'
> 
> ZFS_DBGMSG(zdb):
> spa_import: importing storage, max_txg=-1 (RECOVERY MODE)
> spa_load(storage, config trusted): LOADING
> disk vdev '/dev/diskid/DISK-ZA18YNHS': vdev_validate: failed reading config
> disk vdev '/dev/diskid/DISK-ZA16XZW9': best uberblock found for spa storage. txg
> 3652256
> spa_load(storage, config untrusted): using uberblock with txg=3652256
> vdev_copy_path: vdev 3017550635642283205: path changed from '/dev/ada2' to
> '/dev/diskid/DISK-ZA16XZW9'
> vdev_copy_path: vdev 1070420248480515562: path changed from '/dev/ada3' to
> '/dev/diskid/DISK-ZA18JE4E'
> disk vdev '/dev/diskid/DISK-ZA18YNHS': vdev_validate: failed reading config
> raidz-0 vdev (guid 12743926568081883225): metaslab_init failed [error=122]
> raidz-0 vdev (guid 12743926568081883225): vdev_load: metaslab_init failed
> [error=122]
> spa_load(storage, config trusted): FAILED: vdev_load failed [error=122]
> spa_load(storage, config trusted): UNLOADING
> spa_load(storage, config trusted): spa_load_retry: rewind, max txg: 3652255
> spa_load(storage, config trusted): LOADING
> disk vdev '/dev/diskid/DISK-ZA18YNHS': vdev_validate: failed reading config
> disk vdev '/dev/diskid/DISK-ZA16XZW9': best uberblock found for spa storage. txg
> 3652253
> disk vdev '/dev/diskid/DISK-ZA16XZW9': failed to read label config
> spa_load(storage, config untrusted): using uberblock with txg=3652253
> spa_load(storage, config untrusted): FAILED: label config unavailable
> spa_load(storage, config untrusted): UNLOADING
> 
> Any idea if this can be fixed and how? 3 out of 4 disk the raidz1 are fine.


Just a theory on what could have happened.
Likely there was a pre-existing dormant checksum error for (at least) one of
blocks.  The removed disk probably carried some data that was needed to repair
the the block (e.g. its parity).  Now ZFS detects the checksum error (122), but
it is not able to fix it.  Unfortunately, the affected block seems to belong to
a metaslab spacemap and ZFS is very paranoid about those.  Last time I checked
the code, ZFS marked a vdev as unavailble if it failed to read a spacemap for
it.  I am not sure if the same applies to a read-only import.

So, maybe try to import the pool readonly and take the data elsewhere and then
re-create the pool.


-- 
Andriy Gapon