Skipping tests that are unimplemented in 32-bit emulation

[Alan Somers]

On Tue, Aug 7, 2018 at 9:24 AM, Mark Johnston <markj at freebsd.org> wrote:

> On Tue, Aug 07, 2018 at 09:08:39AM -0600, Alan Somers wrote:
> > On Tue, Aug 7, 2018 at 9:03 AM, Mark Johnston <markj at freebsd.org> wrote:
> >
> > > On Mon, Aug 06, 2018 at 02:18:35PM -0600, Alan Somers wrote:
> > > > On Mon, Aug 6, 2018 at 1:14 PM, Mark Johnston <markj at freebsd.org>
> wrote:
> > > >
> > > > > On Sun, Jul 29, 2018 at 11:23:33AM -0600, Alan Somers wrote:
> > > > > > I recently tried running the i386 test suite in a chroot on an
> amd64
> > > > > > system.  162 tests failed, and 33 were broken.  Some of the
> failures
> > > were
> > > > > > due to system calls that haven't been implemented in 32-bit
> > > emulation.
> > > > > > setfib(2) is an example.  I think it's unlikely that anybody will
> > > ever
> > > > > need
> > > > > > 32-bit emulation for setfib(2), so perhaps we should just skip
> the
> > > test?
> > > > > > What's the best way to do that?  I can come up with two ways:
> > > > > >
> > > > > > 1) At runtime, check the hw.machine sysctl and see if it matches
> some
> > > > > > compile time preprocessor constant.  I don't know what constant
> to
> > > use,
> > > > > > though.  Checking __amd64__ would only work for i386 binaries on
> > > amd64
> > > > > > kernels, and not something else like mips binaries on mips64
> kernels
> > > (I
> > > > > > don't know if we support that, but I don't want to rule it out).
> > > > > >
> > > > > > 2) At buildtime, put an "allowed_architectures=i386" metadata
> > > property
> > > > > into
> > > > > > the Kyuafile for that test program.  This would require support
> in
> > > > > > /usr/share/mk/bsd.test.mk.  It would also require patching Kyua
> > > itself,
> > > > > > because currently "Kyua config" returns the architecture for
> which
> > > it was
> > > > > > built, not the one on which it's running.
> > > > > >
> > > > > > Thoughts?
> > > > >
> > > > > I don't have any particular suggestions, but I'd personally rather
> > > avoid
> > > > > a solution that requires tests to opt-in to running under 32-bit
> > > > > emulation, which I think excludes 2).  I'd be happy to help
> annotate
> > > > > any failing tests as required.  It bugs me that the test suite
> > > currently
> > > > > doesn't cover such relatively complicated functions as
> > > > > freebsd32_copy_msg_out().
> > > >
> > > > I don't think that 2 would necessarily be opt-in, because an
> undefined
> > > > value for allowed_architectures is interpreted as meaning "all".  It
> > > could
> > > > be opt-out instead.  But it could still be a little awkward.  Option
> 1
> > > > could be accomplished for atf-c testcases by comparing the value of
> > > > __LP64__ to a hardcoded list of known 64-bit processors as returned
> by
> > > > uname(3).  But I don't know how to implement 1 for atf-sh programs.
> An
> > > > atf-sh program would need to know the architecture of any binary
> that it
> > > > might invoke.  Is there anything in /etc indicating what
> architecture the
> > > > image was built for?  Should we just use "file /lib/libc.so.*"?
> > >
> > > Could we instead build and install a /usr/tests32 suite on systems that
> > > can support it, and use a top-level Kyuafile and kyua.conf that
> overrides
> > > the "architecture" variable?
> > >
> >
> > It's not enough to build and install the atf-c tests in 32-bit mode.  We
> > would need an entire 32-bit chroot in order to run the atf-sh tests in
> > 32-bit mode.
>
> Oops, right.  For 2), then, couldn't we define test_suite("FreeBSD32")
> in the kyuafiles for the 32-bit test suite and set a
> test_suite.FreeBSD32.compat32=true variable in kyua.conf?  Then a test
> which uses, say, setfib(1) can query that variable and skip if it's set.
>

Ok, but what will set the test_suite.FreeBSD32.compat32 variable?  It can't
be set as part of the image build, because then it will have the wrong
value in jails and chroots.  It needs to be set at runtime.