svn commit: r343543 - head/sbin/bectl/tests

[Kyle Evans]

On Tue, Jan 29, 2019 at 10:55 AM Enji Cooper <yaneurabeya at gmail.com> wrote:
>
>
> > On Jan 29, 2019, at 08:39, Kyle Evans <kevans at freebsd.org> wrote:
> >
> >> On Tue, Jan 29, 2019 at 10:32 AM Enji Cooper <yaneurabeya at gmail.com> wrote:
> >>
> >>
> >>>> On Jan 28, 2019, at 20:10, Kyle Evans <kevans at freebsd.org> wrote:
> >>>>
> >>>> On Mon, Jan 28, 2019 at 10:09 PM Kyle Evans <kevans at freebsd.org> wrote:
> >>>>
> >>>> Author: kevans
> >>>> Date: Tue Jan 29 04:08:49 2019
> >>>> New Revision: 343543
> >>>> URL: https://svnweb.freebsd.org/changeset/base/343543
> >>>>
> >>>> Log:
> >>>> bectl(8) test: Force destroy the zpool in cleanup
> >>>>
> >>>> This is a wild guess as to why bectl tests failed once upon a time in CI,
> >>>> given no apparent way to see a transcript of cleanup routines with Kyua. The
> >>>> bectl tests construct a new, clean zpool for every test. The failure
> >>>> indicated was because of a mount that was leftover from a previous test, but
> >>>> the previous test had succeeded so it's not clear how the mount remained
> >>>> leftover unless the `zpool get health ${pool}` had somehow failed.
> >>>>
> >>>
> >>> I left out: the tests are supposed to be constructed to clean up any
> >>> mounts that were left over in the course of the test, hence the
> >>> assumption that the failure lies in the cleanup.
> >>
> >> Hi Kyle,
> >>
> >> The tests use a deterministic zpool name defined locally (not globally), and will only destroy the zpool if “zpool get health” succeeds.
> >>
> >> The tests will work the first time (when the zpool doesn’t exist), but I believe they’re actually introducing nondeterminism by accident. I will propose a fix for this.
> >>
> >> There’s a way to decipher why things failed from /var/log/messages and kyua output. It’s just nontrivial to those who don’t know what to look for. Can you please provide a failing ci run?
> >>
> >
> > Hi,
> >
> > Thanks! Any help is appreciated -- my inquiry to -testing@ in response
> > to the weekly report where the failing test [1] was mentioned has been
> > met with silence. The failure is consistent in the i386-test job, but
> > amd64-test sees nothing of the sort (and neither does my local
> > testing).
>
> Thanks for the reminder to rejoin that list.
>
> > [1] https://ci.freebsd.org/job/FreeBSD-stable-12-i386-test/426/testReport/sbin.bectl/bectl_test/bectl_mount/
>
> It’s pretty obvious from the above run what’s going on from the output. A similarly named zpool (same prefix; suffixed with a 2) is being matched by grep, even though the test sets up one with a suffix in the preceding steps.
>

Right- this is the part that makes sense to me. =) The 'default2' BE
is clearly a leftover from a previous test, but it should've been
unmounted in the process and the zpool cleaned up.

What's not clear is how the cleanup routine is failing; the `zpool get
health` shouldn't have any reason to fail (though I can't test it
here, apparently =() and `zpool destroy -f` should've unmounted this
BE even if that hadn't happened throughout the course of the test, but
the failure remains (even if probably incorrectly fixed) following the
above commit.

Setting the zpool name globally with a suffix unique to this test
would likely be a great idea -- I think there's something else going
on as well, though.

Thanks,

Kyle Evans