Oh no, what have I done...

[Paul Kraus]

> On Aug 12, 2017, at 10:48 PM, Chris Ross <cross+freebsd at distal.com> wrote:
> 
>> On Aug 12, 2017, at 22:23 , Paul Kraus <paul at kraus-haus.org> wrote:

>> The same disk. ZFS is looking for the labels it writes (4x for redundancy; 2 at the beginning of the disk and 2 at the end; it only _needs_ to find one of them) to identify the drive to include it in the zpool. If that disk no longer contains the ZFS labels then you are probably out of luck.
> 
>  Okay.  Well, the work I had done on the other disk to “wipe” it only wiped the beginning.  So I appear to have gotten lucky that ZFS writes labels at the end, and also learned that my intent of wiping it was insufficient.  In this case, to my benefit.  I was able to bring the zpool back online.

You got luck (and in this case you win :-)

> 
>> You _may_, and this is a very, very long shot, be able to force an import to a TXG (transaction group) _before_ you added the lone drive vdev.
> 
>  I’m curious to learn more about this.  Had I inserted an geometrically identical disk, without the labels ZFS was able to find, and the above was my option.  How could I have done that?  And, now that I have it up again, am I moving farther away from that possibility?

Using zdb you can examine the TXGs and there is a way to force an import to a certain TXG (at least under Illumos there is, I assume the FBSD ZFS code is the same). I do not remember the specific procedure, but it went by on one of the ZFS mailing lists a few years ago. The goal at the time was not to deal with a changed pool configuration but a series of writes that corrupted something (possibly due to a feature being turned on at a certain point in time). You would lose all data written after the TXG you force the import to.

Looking at the manpages for zpool and zdb, it looks like doing this has gotten easier with `zdb -e -F` … operate on exported pool and rollback TXG until the pool is importable … but I’m sure there are limits :-)

Spend time with the zdb manpage and remember that many of these functions were added due to someone needing to do something “inappropriate” to a pool to recover some data :-)

> 
>> 
>>> I’ll try putting a disk with the same layout in, and see how that goes.  That was a thought I’d had.
>> 
>> Won’t work, without the ZFS labels it will not be seen as the missing drive.
> 
>  K.  What, if it hadn’t found labels I’d fail to clear, would it have done with it?  Just considered it irrelevant, and still refused to bring the pool in much the same was as I’d been trying with the disk forcibly removed?

ZFS would have considered the new drive as just that and the pool would still be un-importable due to a missing top level vdev.

>  Thanks all.  I’d like to learn more about what to do in this situation, but I appear to have at least gotten to the point where I can create a backup, which should allow me to destroy and create a pool more to my actual intent.

Keep in mind are that you cannot add devices for capacity _within_ a vdev only to add mirrors, you add capacity to a pool by _adding_ vdevs. As you learned, all of the top level vdevs do not have to be the same type (mirror, radizn), but with different types of vdevs it is almost impossible to predict the performance of the pool. These days I generally only use raidz2 for backup data or long term storage, I use mirrors (2 or 3 way) for production data. This gets me better performance and easier scalability.

ZFS is very good at getting you access to your data as long as it can find enough of each top level vdev to bring them all on line. I once had a hardware config with limitations, so I designed my pools around that and _when_ the external disk chassis went offline I lost performance but no data.

When trying to recover a ZFS pool, never overwrite any drives that may have been part of the pool as you may need them to bring the pool back online.