Schedule for releases

[Alexander Leidinger]

On Wed, 29 Dec 2010 03:17:41 -0800 Garrett Cooper <gcooper at FreeBSD.org>
wrote:

> On Tue, Dec 28, 2010 at 3:01 PM, Alexander Leidinger
> <Alexander at leidinger.net> wrote:

> >> We still lack the parts that would tell us something in the last
> >> week or last 24 hours caused a regression that made my
> >> TCP/NFS/ZFS/UFS/<you name it> n% slower.  Kris had been doing a
> >> good job in the past but as time shows we need more people,
> >> different setups, ...
> >
> > We do not lack the parts, we lack someone to take the parts and get
> > them up and running. See:
> >  http://www.mail-archive.com/freebsd-performance@freebsd.org/msg02819.html
> >  http://www.mail-archive.com/freebsd-performance@freebsd.org/msg02821.html
> 
> No: according to Erik in the first link, the parts were never fully
> integrated and the project as a whole was never completed due to lack
> of hardware and time. Some bits are also just directions as well as

I understand this link (and my maybe flaky memory when he presented
his work the first time) as he didn't had the time and hardware to set
up the automatic benchmarking suite to run periodically "forever". And
what you call directions looks to me like documentation.

As Erik is participating in this thread, I let the last word for him. :)

> Erik says in his reply to the thread. After seeing enough folks
> prematurely claim victory within my own company, I know enough that
> nothing is complete unless it works (to some basic degree, i.e. it can
> be replicated by someone other than the original author of the
> infrastructure or code using the directions provided from scratch).

I agree.

Erik can you make your work available for download? If not, I offer
webspace (either on my private site or in my area of the FreeBSD owned
webspace) to make it possible to donwload it.

> >> It's not only "compiles", "boots", but also the formerly in this
> >> thread mentioned "works correctly" and in addition to that the
> >> "works well as expected" or "works better than before" -
> >> hopefully;).
> >
> > Maybe this could also be used to run the regression tests as one of
> > the benchmarks. If yes: As Robert mentioned, we can not go and tell
> > to run them all in one command (ATM), but we could have each of
> > them as a different benchmark.
> 
> Regression tests should be stable. Regression tests shouldn't crash
> targets (in general). Benchmarks and stress tests can do that. We need

I was referring to his explanation that there is not central place to
kick off all of them. I agree that the should be stable, but in case of
a regression a crash may be a possible failure scenario (which can
then be fixed by fixing the regression).

> IMO we should have something like:
> 
> make checkworld
> make checkkernel

I agree that this would be nice. I think this is one possible
implemenation of what Robert whas referring to when he told that there
is no central place to start all the regression tests.

> that would go off and run functional regression tests, and something
> like:
> 
> make stressworld
> make stresskernel
> 
> that would to similar for benchmarks as well. Having stuff tied

Depending on the benchmark you need some kind of setup which can not be
done by such targets (if we assume that this shall be self-contained
and possible just by running those commands instead of doing some
setup of 3rd part software like databases or webservers before).

Bye,
Alexander.