Running tests as a developer prior to commit

[Allan Jude]

On 2015-02-23 21:37, Craig Rodrigues wrote:
> On Mon, Feb 23, 2015 at 2:19 PM, Ed Maste <emaste at freebsd.org> wrote:
> 
>>
>> It'd be even better if we can facilitate simple runs of a
>> suitable subset of tests prior to a commit. It looks like "make test"
>> is close, despite the warnings it emits about being experimental. Is
>> there a path we can take to support it, for at least a common case of
>> developing and testing on a FreeBSD-CURRENT install?
>>
> 
> On the surface, your request seems simple.
> However, I have seen the same request asked in multiple companies
> I have worked at, and it is not so simple.
> 
> If someone touches an arbitrary piece of code in FreeBSD,
> what are the subset of tests that need to be run to validate the change?
> Some changes are very simple, and it is quite easy to extrapolate which
> tests need to be run.
> 
> However, some changes are more subtle, and it is not so obvious which
> tests are affected.
> 
> Having knowledge about what changes affect what unit tests requires almost
> an expert system to be built.
> 
> The other suggestion I have seen in other companies is to require developers
> to run all unit tests for every change before they commit.  This works to
> some
> extent, because in a company you can force this behavior.
> 
> For FreeBSD, I think this will fail.  We already have over 3000 tests under
> /usr/tests.
> It's not that hard to run them with "cd /usr/tests && kyua test".
> However, requiring FreeBSD developers to run these tests
> for every change they do is a big change in how they submit code to FreeBSD.
> This will probably result in rebellion.
> 
> I would like to see a compromise solution.  I would like an optional
> workflow,
> where someone does a pull request on the FreeBSD repository in GitHub,
> which then
> kicks off automation which builds an image, runs the kyua tests, and
> reports the results.
> 
> After seeing these test results, the developer would still be responsible
> for manually
> committing the change to SVN.
> 
> People who are OK with using GitHub can follow this workflow, and opt-in to
> the tests.
> People who hate git and GitHub can ignore it, and continue to commit
> directly to SVN
> as they do today.
> 
> There are a whole lot of automation libraries built on top of Jenkins and
> GitHub
> which can facilitate this type of workflow.  Projects like Mac Homebrew
> have very
> sophisticated workflows which kick off tests in response to pull requests.
> 
> Someone in FreeBSD would need to build out this test infrastructure.  It's
> doable, but
> a lot of work.  It might be nice to make a Google Summer of Code project
> out of this.
> 
> --
> Craig
> _______________________________________________
> freebsd-testing at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-testing
> To unsubscribe, send any mail to "freebsd-testing-unsubscribe at freebsd.org"
> 

Doesn't phabricator have some hooks to launch tests of a "proposed"
patch? This could help catch the errors before they are committed

-- 
Allan Jude

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 834 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-testing/attachments/20150223/91db7a35/attachment.sig>