svn commit: r208954 - in head/contrib/llvm: . docs test tools/clang tools/clang/docs tools/clang/test website

[Garrett Cooper]

On Jun 9, 2010, at 3:20 PM, Pawel Worach wrote:

> On Jun 9, 2010, at 23:58, Garrett Cooper wrote:
> 
>> On Wed, Jun 9, 2010 at 12:04 PM, Ed Schouten <ed at 80386.nl> wrote:
>>> Hi Garrett,
>>> 
>>> * Garrett Cooper <yanefbsd at gmail.com> wrote:
>>>> Why strip test? This might actually be helpful for folks trying to
>>>> evaluate whether or not they should upgrade to newer versions of
>>>> clang.
>>> 
>>> The testsuite can be checked out separately from the vendor space. It
>>> will account for about 50 MB of additional disk space usage.
>> 
>> Yes, but the tests can change at any point in time and might not
>> reflect the code stored in the repository.
>> 
>> Could a note at least be provided, or a port maintained to help others
>> (say vendors) who have to qualify FreeBSD that this is the particular
>> version that needs to be picked up from <blah> in order to test our
>> shiny new compiler? That would make things considerably easier to work
>> with, as I've worked as QA in a Linux shop in the past, where a lot of
>> Linux provided packages that we had in the tree did not have the
>> associated test code, and as QA that made my job a pain to deal with.
>> 
>> Just a small file that stated where and/or how to obtain things would
>> make this a lot easier.
>> 
>> I would think that developers would also like the same to see whether
>> or not there are any particular known issues with clang/llvm.
>> 
> 
> We have a buildbot running here[1] that has the goal of serving as a qualification system.
> 
> It does a nightly build of llvm/clang trunk in a self-host config, runs the test suite in both stages and then builds freebsd trunk world+kernel and boots the result.

Yes, but how much is tested, and what code paths are tested in the compiled kernel? Do you have all of the devices required to test all of the drivers and extensive tests to exercise everything? Probably not; it sounds like what you have setup wise is a smoke test box -- which is good but it's not comprehensive enough to catch all issues...

That also doesn't test out i386, or powerpc.

Ideally the project should have a cluster of dedicated machines for regression testing available in a colo somewhere where someone can load images on the machine via netboot, or install, then execute a set of functional and/or performance tests to gauge whether or not the code has functional or performance regressions.

Don't get me wrong, what you guys have is good -- but it's just not 100% convincing for someone making a switchover to a newer toolchain, because of all of the different variables that might be out there with all of the code in FreeBSD, as well as interactions with compiled code, and 3rd party software as well.

But this is just the beginning of clang on FreeBSD and the project in and of itself has a long way to go. Adding the bits so that other groups could help evaluate, add tests, etc to ensure that clang/llvm works as expected is what we should be focusing on, not just runtime tests and `works for me' scenarios.

Thanks,
-Garrett