regression-test

[Kris Kennaway]

Wesley Shields wrote:
> On Thu, Jun 19, 2008 at 08:54:17AM -0700, pfgshield-freebsd at yahoo.com wrote:
>> Christian Weisgerber wrote:
>> ...
>>> In particular, running tests of what duration would be considered
>>> appropriate?  E.g., building audio/flac takes a minute here, running
>>> the tests sixteen, which I suspect would be excessive.
>> FWIW, I started adding those regression-test targets to many ports
>> because they are extremely useful: while trying to update to the
>> latest coreutils I found everything apparently built but everything
>> was actually broken. Now I am finding bugs in some ports that I
>> thought were fine.
> 
> Which is why I would recommend wrapping them in checks for
> MAINTAINER_MODE or something similar.  This way you, as the maintainer,
> can turn them on when you want but they don't affect anything else.

Our experience has shown that it is very common for tests to succeed 
when run by the maintainer, but to fail in the package cluster 
environment because of differences in the environment, configuration, 
default assumptions, etc.  i.e. the regression tests found bugs, which 
is the entire point.

We *want* regression tests to be available unconditionally -- that is 
why I added the hook to the package build process to run them if 
present.  Please don't hack around this because you think you're doing 
us a favour :-)

Kris