About ports QA by 'port test'

[chukharev at mail.ru]

Recently I've been trying to run some additional tests during updating
the ports I have installed on my computer. That means I select a number
of oldest installed ports (using dates of directories in /var/db/pkg/),
and for each of them run 'port test' (ports-mgmt/porttools) and then
'portupgrade -f'.

FYI, the 'port test' uses portlint, builds in a different place, installs
into a different place, packages, then de-installs and checks for left
files. No jail, no chroot.

  From what I have seen till now, about 25% of the ports do not pass the
test, either for fatal errors from portlint or due to errors exhibited
because of PREFIX and PKG_DBDIR variables. And this is done only for
the ports I have successfully installed on my system for some reason
not connected to the QA purposes.

I do not want to trouble the port maintainers with direct e-mails.

I've been posting the results (and the original description I'm modifying
now) to freebsd-qa@ for a while, but that turned out to be of some burden
and also I received there no feedback at all.
Therefore I set a webpage at my day-job where I will copy the results.
I hope I will not need to keep the site for long, if running 'port test'
is found useful, it should be added into QAT or tinderbox, I think. Though
without jail it is faster I guess.

The URL: http://kemia.me.tut.fi/~chu/FreeBSD.port_tests/
Current statistics:

Ports having fatal errors:             109 (21 %)
Ports having warnings:                 216 (43 %)
Ports without fatal errors:            391 (78 %)
Ports having warnings or fatal errors: 325 (65 %)
Totally tested ports:       500

Additionally to the above stats, java/jdk16 is interactive and hangs on input
despite of BATCH=yes in /etc/make.conf.

-- 
Vladimir Chukharev
Tampere University of Technology