ZFS pools in "trouble"

[Alan Somers]

On Sat, Mar 14, 2020 at 9:14 AM Andriy Gapon <avg at freebsd.org> wrote:

> On 14/03/2020 13:00, Willem Jan Withagen wrote:
> > On 27-2-2020 09:11, Andriy Gapon wrote:
> >> On 26/02/2020 19:09, Willem Jan Withagen wrote:
> >>> Hi,
> >>>
> >>> I'm using my pools in perhaps a rather awkward way as underlying
> storage for my
> >>> ceph cluster:
> >>>      1 disk per pool, with log and cache on SSD
> >>>
> >>> For one reason or another one of the servers has crashed ad does not
> really want
> >>> to read several of the pools:
> >>> ----
> >>>    pool: osd_2
> >>>   state: UNAVAIL
> >>> Assertion failed: (reason == ZPOOL_STATUS_OK), file
> >>> /usr/src/cddl/contrib/opensolaris/cmd/zpool/zpool_main.c, line 5098.
> >>> Abort (core dumped)
> >>> ----
> >>>
> >>> The code there is like:
> >>> ----
> >>>          default:
> >>>                  /*
> >>>                   * The remaining errors can't actually be generated,
> yet.
> >>>                   */
> >>>                  assert(reason == ZPOOL_STATUS_OK);
> >>>
> >>> ----
> >>> And this on already 3 disks.
> >>> Running:
> >>> FreeBSD 12.1-STABLE (GENERIC) #0 r355208M: Fri Nov 29 10:43:47 CET 2019
> >>>
> >>> Now this is a test cluster, so no harm there in matters of data loss.
> >>> And the ceph cluster probably can rebuild everything if I do not lose
> too many
> >>> disk.
> >>>
> >>> But the problem also lies in the fact that not all disk are recognized
> by the
> >>> kernel, and not all disk end up mounted. So I need to remove a pool
> first to get
> >>> more disks online.
> >>>
> >>> Is there anything I can do the get them back online?
> >>> Or is this a lost cause?
> >> Depends on what 'reason' is.
> >> I mean the value of the variable.
> >
> > I ran into the same problem. Even though I deleted the zpool in error.
> >
> > Ao I augmented this code with a pringtf
> >
> > Error: Reason not found: 5
>
> It seems that 5 is ZPOOL_STATUS_BAD_GUID_SUM and there is a discrepancy
> between
> what the code in status_callback() expects and what actually happens.
> Looks like check_status() can actually return ZPOOL_STATUS_BAD_GUID_SUM:
>         /*
>          * Check that the config is complete.
>          */
>         if (vs->vs_state == VDEV_STATE_CANT_OPEN &&
>             vs->vs_aux == VDEV_AUX_BAD_GUID_SUM)
>                 return (ZPOOL_STATUS_BAD_GUID_SUM);
>
> I think that VDEV_AUX_BAD_GUID_SUM typically means that a device is
> missing from
> the pool.  E.g., a log device.  Or there is some other discrepancy between
> expected pool vdevs and found pool vdevs.
>

This situation can also arise if you remove a device from the pool (either
physically, or because the device failed), then you change the pool's
configuration, such as by replacing the missing vdev, then you reinsert the
original device and reboot.  ZFS may pick up the old device's label, and
the guid sums won't match.  Another way you can get this error is by
creating a ZFS pool on a disk, then repartitioning the disk and creating a
new pool.  It's possible that "zpool import" will see the old, invalid
label before it sees the newer valid label.  I've even run into a situation
where "zpool import" worked fine, but the bootloader failed, because they
interrogate labels in a slightly different way.
-Alan