restore zfs pool using original devices?

[Ruben van Staveren]

Hi list,

Recently I decided to grow my zfs pool from 1TB to 6TB by replacing my 1TB dell rebranded toshiba disk with an
HGST Deskstar NAS 6 TB one. system was running 10.2

to make things easier, I decided to do a zpool replace on both the unencrypted boot and encrypted (as per https://www.dan.me.uk/blog/2012/05/06/full-disk-encryption-with-zfs-root-for-freebsd-9-x/) system pool

zpool replace system gpt/boot diskid/DISK-NAHRVZDXp2
zpool replace system gpt/system.eli diskid/DISK-NAHRVZDXp3.eli

The system ran fine for a while until I rebooted it:

https://ruben.is.verweg.com/stuff/IMG_5734.jpg

a panic during mounting root

panic: solaris assert: nvlist_lookup_uint64(configs[i], ZPOOL_CONFIG_POOL_TXG, &txg) == 0: file src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:4040

I tried to recover diskid/DISK-NAHRVZDXp2 by creating a new boot pool on a rescue nanobsd image, so I could recover the encryption keys. No I didn’t make an off system backup of those with zpool create -R /path/to/temp/root-for-boot zboot /dev/diskid/DISK-NAHRVZDXp2.

first it complained the disk was part of the original boot pool, so I used -f. upon inspection I found the disk to be empty, so it looks like that action destroyed some important metadata. should it do that actually?

I wanted to restore that data so I used dd if=/dev/gpt/boot of=/dev/diskid/DISK-NAHRVZDXp2 bs=64k but the contents is not restored.
a cat of /dev/gpt/boot show various data is still there, but not seen anymore by zfs.

given zfs replace actually involves a temporarily mirror in where the old vdev is removed, it should be pretty much possible to re attach this vdev to any pool to see its contents again ?

what are my chances to create a functional new pool using old devices?

Best Regards,
    Ruben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 203 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20160202/69094dc4/attachment.sig>