Non-stable tests

Olivier Cochard-Labbé olivier at
Wed May 1 16:41:00 UTC 2019

On our CI (Jenkins running the full kyua test suite on a bhyve VM), some
tests are sometimes failing without a valid cause.
So I had to run a second time the kyua test suite to confirm it's a "real"
regression, and the second time I didn't meet any problem.
The last 2 tests that meet this problem are:
- sbin.devd.client_test.seqpacket, failed with "Test case body timed out"
- lib.libc.sys.getitimer_test.setitimer_old, failed with "setitimer(2) did
not return remaining time, failed with"

I've tried to rerun these tests a thousand times, but I didn't reach to
reproduce the problem: they've succeeded each time after.

So I wonder if there is a problem about "non stable" tests ?

I've done a rapid search on and I've found an example.
In this series: (r344839) (r344840) (r344843)

=> with job 10392, the test "lib.libarchive.functional_test.test_fuzz_zip"
failed, but there is no change in the code regarding libarchive during
these 3 commits revision period (
So why it failed once here ?
This is the same symptom I meet on my platform too.



More information about the freebsd-testing mailing list