RFC: Suggesting ZFS "best practices" in FreeBSD

[claudiu vasadi]

On Tue, Jan 22, 2013 at 4:36 PM, Borja Marcos <borjam at sarenet.es> wrote:

>
> On Jan 22, 2013, at 4:04 PM, Warren Block wrote:
>
> > I'm a proponent of using various types of labels, but my impression
> after a recent experience was that ZFS metadata was enough to identify the
> drives even if they were moved around.  That is, ZFS bare metadata on a
> drive with no other partitioning or labels.
> >
> > Is that incorrect?
>
> I'm afraid it's inconvenient unless you enjoy reboots ;)
>
> This is a patologic and likely example I just demonstrated to a friend.
>
> We were testing a new server with 12 hard disks and a proper HBA.
>
> The disks are, unspririsingly, da0-da11. There is a da12 used (for now)
> for the OS, so that there's no problem to create and destroy pools at
> leisure.
>
> My friend had created a pool with two raidz vdevs nothing rocket science.
> da0-5, da6-11.
>
> So, we were doing some tests and I've pulled one of the disks. Nothing
> special, ZFS recovers nicely.
>
> Now it comes the fun part.
>
> I reboot the machine with the missing disk.
>
> What happens now?
>
> I had pulled da4 I think. So, disks with an ID > 4 have been renamed to N
>  - 1. da5 became da4, da6 became da5, da7 became da6... and, critically,
> da12 became da11.
>
> The reboot begun by failing to mount the root filesystem, but that one is
> trivial. Just tell the kernel where it is now (da11) and it boots happily.
>
> Now, we have a degraded pool with a missing disk (da4) and a da4 that
> previously was da5. It works of course, but in degraded state.
>
> OK, we found a replacement disk, and we plugged it. It became, guess! Yes,
> da12.
>
> Now: I cannot "online" da4, because it exists. I cannot online da12
>  because it didn't belong to the pool. I cannot replace da4 with da12,
> because it is there.
>
> Now that I think of it, in this case:
> 15896790606386444480  OFFLINE      0     0     0  was /dev/da4
>
> Is it possible to say zpool replace 15896790606386444480 da12? I haven't
> tried it.
>
> Anyway, seems to be a bit confusing. The logical, albeit cumbersome
> approach is to reboot the machine with the new da4 in place, and after
> rebooting, onlining or replacing.
>
> Using names prevents this kind of confusion.
>
>
>
>
>
Same thing happened to me on a production server that crashed. Sometimes,
it;s not easy to reboot again simply because you need to insert a disk.
This, of course, make s the hot-swappable capability of the hardware,
useless.

-- 
Best regards,
Claudiu Vasadi