[CFT/RFC]: refactor bsd.prog.mk to understand multiple programs instead of a singular program

[Simon J. Gerraty]

>> Not to mention the fact that bsd.prog.mk goes from being relatively
>> simple, to unspeakably hard to read, and all for rather limited =
>return.

This btw I think is the more important issue.
I was looking at bsd.prog.mk in netbsd the other day.
It has no business being that complex.

>Getting back to the idea at hand, the enhancement goal was to get the =
>testcases to live [and optionally be built/installed] with the source =
>code to avoid bitrot; I know this isn't the current NetBSD design, but =
>this is what was requested be done by gnn, marcel, and mdf, and I agree. =

I agree, that's what we do too.

>It order to do this, I need to be able to build multiple programs so =
>things are as transparent. On the flipside, bsd.prog[s].mk can live on =

We put the test cases in a subdir of the lib/prog
This has multiple benefits, and eliminates any impact on the normal
build of said libs/progs.

Also a number of our teams find it necessary to create mock classes etc to
adequately test their libs.  Keeping all this in a tests/ subdir makes
it easy, to keep things neat & up to date, and ensure that the tests
pass.  Trying to do all that in the same dir as the lib would be ugly.

>> FWIW we use progs.mk (as bsd.progs.mk) from
>> ftp://ftp.netbsd.org/pub/NetBSD/misc/sjg/mk-*.tar.gz=20
>> It isn't ideal, but it certainly avoids a lot of churn and complexity
>> for what is essentially a corner case.
>
>This requires pulling bmake into FreeBSD proper in order for things to =
>function the last I checked as bsd.prog.mk depends upon bmake directives =

This is already happening ;-)

>Ideally however, I would like doing this compared to running a custom =
>build infrastructure, but (as you probably know) this requires =
>rototilling the current FreeBSD build system to a large degree =

Actually building FreeBSD-current using bmake requires only modest changes.

>> We have an atf.test.mk which leverages that.
>> It also handles automatically running the tests if building for the
>> host. This allows us to fail the build if any unit-tests do not pass.
>
>Hmm=85 that wasn't really the end-goal of bsd.test.mk based on my =
>reading,

I know, but it is a very useful thing to be able to do.
You can add knobs to controll such things of course.

>bsd.test.mk in NetBSD isn't completely test-agnostic, but it isn't =
>entirely ATF centric either. I'm trying not to stray too far from NetBSD =
>in this area because there really isn't any reason to do so. FWIW I'd =

Sure - we imported ATF directly from NetBSD to minimize the churn
and not had to change much at all.
bsd.test.mk was a point worth diverging on though.

>like to see other test frameworks (lua, unittest//nose, etc) just snap =
>into bsd.test.mk as easily as possible as it would make some groups =

Yes, but making one test.mk handle potentially lots of frameworks
will get rather ugly quite quickly.

Better to add per-framework .mk files, and perhaps have them include a
bsd.test.mk which only handles high level logic rather than the details
of the frameworks.

That's laregly why we didn't use bsd.test.mk

--sjg