Re: FreeBSD-main-amd64-test - Build #19523 - Unstable

On Thu, 14 Oct 2021 at 18:30, <jenkins-admin@freebsd.org> wrote:
>
> FreeBSD-main-amd64-test - Build #19523 (86929782cfab2664010a31979e9e3c0626721ebb) - Unstable
>
> Build information: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19523/
> Full change log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19523/changes
> Full build log: https://ci.FreeBSD.org/job/FreeBSD-main-amd64-test/19523/console
>
> Status explanation:
> "Unstable" - some tests are suspected being broken by the following changes
> "Still Unstable" - the failing test cases have not been fixed by these
>                    following changes and this is a notification to note that
>                    these changes have not been fully tested by the CI system
>
> Change summaries:
> (Those commits are likely but not certainly responsible)
>
> 1adebca1fc9f6def501f7a5c2b2bde5fe9049477 by kib:
> Style
>
> 86929782cfab2664010a31979e9e3c0626721ebb by kib:
> Fix typo in comment
>
>
>
> The failed test cases:
>
> 1 tests failed.
> FAILED:  sys.netpfil.pf.killstate.v6
>
> Error Message:
> atf-check failed; see the output of the test for details

I wonder how we can make the test results more accessible, so that a
committer can quickly determine that this is most likely a false
positive unrelated to their change (as here). Perhaps a link to the
individual test results, i.e.
https://ci.freebsd.org/job/FreeBSD-main-amd64-test/19523/testReport/junit/sys.netpfil.pf/killstate/v6/?

Although it would be really nice if we could include a useful line or
two from the test output I think it's unfeasible to programmatically
find the appropriate information, and it may not exist anyway. (In
this case the best we can probably extract is "Fail: incorrect exit
status: 1, expected: 0" which is not really any help.)