100% repeatability?

[Alan Somers]

On Thu, Feb 27, 2014 at 1:17 AM, Peter Holm <peter at holm.cc> wrote:
> I looked at creating a test scenario for the following piece of code
> from src/sys/kern/kern_fork.c:
>
>    881          if ((nprocs >= maxproc - 10 && priv_check_cred(td->td_ucred,
>    882              PRIV_MAXPROC, 0) != 0) || nprocs >= maxproc) {
>    883                  error = EAGAIN;
>    884                  goto fail;
>    885          }
>
> I came up with a test scenario for this:
>
> http://people.freebsd.org/~pho/kern_fork_test.c
>
> But it is not 100% repeatable, as I would expect many other kernel
> tests will be.
> The test case runs as expected, until for example root decides to do a
> parallel buildworld.
> Is this a show stopper?

First of all, let me say thank you for writing ATF tests.  It's great
that the community is starting to use it.  As for the test:

1) nlistf and memf should be explicitly initialized, and may as well
be private variables in getprocs().

2) global variables are unfortunate.  Would it be possible for
getprocs() to return maxprocs and nprocs as
return arguments instead of global variables?  Alternatively, it could
return (maxprocs - nprocs), since that's the only way they're used.

3) It's confusing to have a testcase with the same name as a global
variable.  It doesn't confuse the compiler, since ATF_TC() and friends
mangle the name.  But it does confuse cscope and humans.

4) Regarding repeatability: non-repeatability is very very bad.  It
leads to spurious failures appearing in kyua's results, which cause
casual testers to ignore the entire suite.  If they come to expect a
few spurious failures, then they'll ignore real failures.  Serious
testers learn to ignore the intermittently failing tests, which isn't
good either.  It simply takes too much effort to debug each and every
failure; people won't do it.  I would not commit a test that isn't
repeatable.

If nprocs and maxproc were jail-specific variables, then you could
simply run this test inside of a jail.  However, they look global to
me.  I think that the only good way to test the feature would be to
run the test inside of a private bhyve instance with an extremely
minimal set of services.  That way you could ensure that no unexpected
processes would be started.  It would be much more work, though.  You
would also need a way to communicate to the guest OS.  Perhaps an
emulated serial port?

-Alan