Re: git: 3fb03fb35b20 - main - [tests] switch GoogleTest tests to `googletest` interface

> On Oct 30, 2026, at 2:43 PM, Mark Johnston <markj@freebsd.org> wrote:
> 
> On Fri, Oct 30, 2026 at 01:54:22PM -0700, Enji Cooper (yaneurabeya) wrote:
>> 
>>> On Oct 29, 2026, at 7:20 PM, Mark Johnston <markj@FreeBSD.org> wrote:
>>> 
>>> On Thu, Oct 29, 2026 at 11:32:53PM +0000, Enji Cooper wrote:
>>>> The branch main has been updated by ngie:
>>>> 
>>>> URL: https://cgit.FreeBSD.org/src/commit/?id=3fb03fb35b20d259e0a3a07fc48f2ecc3c5da6e8
>>>> 
>>>> commit 3fb03fb35b20d259e0a3a07fc48f2ecc3c5da6e8
>>>> Author:     Enji Cooper <ngie@FreeBSD.org>
>>>> AuthorDate: 2026-10-07 18:53:15 +0000
>>>> Commit:     Enji Cooper <ngie@FreeBSD.org>
>>>> CommitDate: 2026-10-29 23:32:11 +0000
>>>> 
>>>>   [tests] switch GoogleTest tests to `googletest` interface
>>> 
>>> Please follow the guidelines for commit log messages:
>>> https://docs.freebsd.org/en/articles/committers-guide/#_prefix_the_subject_line_with_a_component_if_applicable
>>> 
>>>>   The new `googletest` kyua test interface--added in 0.15.0 and fixed in
>>>>   0.15.1--allows tests to be executed and handled in a more fine-grained
>>>>   manner than the way it was prior to this commit.
>>>> 
>>>>   This change deletes the `capsicum-test` ATF wrapper script,
>>>>   `functional.sh`, and all supporting logic in `googletest.test.mk` as
>>>>   well as this change makes that logic unnecessary.
>>>> 
>>>>   MFC after:      1 month
>>>>   MFC with:       b68969979d6b ("contrib/kyua: update to 0.15.1")
>>>> ---
>>>> share/mk/googletest.test.mk      | 11 +-------
>>>> tests/sys/capsicum/Makefile      |  1 -
>>>> tests/sys/capsicum/functional.sh | 60 ----------------------------------------
>>>> 3 files changed, 1 insertion(+), 71 deletions(-)
>>>> 
>>>> diff --git a/share/mk/googletest.test.mk b/share/mk/googletest.test.mk
>>>> index 026f6dcacce3..b21576112a78 100644
>>>> --- a/share/mk/googletest.test.mk
>>>> +++ b/share/mk/googletest.test.mk
>>>> @@ -6,10 +6,6 @@
>>>> # GoogleTest is a C++ test framework, thus, it does not describe/articulate how
>>>> # to write tests in other languages, e.g., C or shell, unlike the ATF, plain,
>>>> # and TAP raw test interfaces.
>>>> -#
>>>> -# For now this is a thin wrapper around the `plain` test interface, but in the
>>>> -# future this will rely on a newer version of kyua which will integrate in
>>>> -# GoogleTest support.
>>>> 
>>>> .if !target(__<bsd.test.mk>__)
>>>> .error googletest.test.mk cannot be included directly.
>>>> @@ -34,13 +30,8 @@ BINDIR.${_T}= ${TESTSDIR}
>>>> CXXFLAGS.${_T}+= ${GTESTS_CXXFLAGS}
>>>> MAN.${_T}?= # empty
>>>> SRCS.${_T}?= ${_T}.cc
>>>> -.if !empty(GTESTS_WRAPPER_SH.${_T})
>>>> -# A stopgap/workaround to let kyua execute test case one by one
>>>> -ATF_TESTS_SH+= ${GTESTS_WRAPPER_SH.${_T}}
>>>> -.else
>>>> _TESTS+= ${_T}
>>>> -TEST_INTERFACE.${_T}= plain
>>>> -.endif
>>>> +TEST_INTERFACE.${_T}= googletest
>>>> 
>>>> .endfor
>>>> .endif
>>>> diff --git a/tests/sys/capsicum/Makefile b/tests/sys/capsicum/Makefile
>>>> index a8f72d399af9..788fa9f9394c 100644
>>>> --- a/tests/sys/capsicum/Makefile
>>>> +++ b/tests/sys/capsicum/Makefile
>>>> @@ -10,7 +10,6 @@ CFLAGS+= -I${SRCTOP}/tests
>>>> .if ${MK_GOOGLETEST} != no
>>>> 
>>>> GTESTS+= capsicum-test
>>>> -GTESTS_WRAPPER_SH.capsicum-test= functional
>>>> # This test script runs the same test suite twice, once as root and once as an
>>>> # unprivileged user.  Serialize them since some tests access global namespaces,
>>>> # e.g., mqueuefs, and can trample on each other.
>>> 
>>> Below this there are some TEST_METADATA.functional configuration
>>> variables still set.
>> 
>> I’ll take a look at this.
>> 
>>> After this commit I see a whole bunch of newly skipped tests, it looks
>>> like a regression:
>> 
>> …
>> 
>> It’s not. The problem is that prerequisites aren’t being met or the test cases were skipped previously, but not counted as skipped.
>> 
>> Prior to this change all GoogleTest tests were run as “plain” interface test programs, which had a binary notion of pass/fail: did the program exit with 0? If so the test passed, else it failed.
>> 
>> Using ` lib/googletest/gtest_main/googletest-printers-test:PrintStlContainerTest.StdSpan` as an example. Before (from https://ci.freebsd.org/job/FreeBSD-main-amd64-test/29832/testReport/lib.googletest.gtest_main/googletest-printers-test/ ):
>> ```
>> [ SKIPPED ] 1 test, listed below:
>> [ SKIPPED ] PrintStlContainerTest.StdSpan
>> ```
>> The testcase was being skipped - the suite as a whole exited as 0, so the suite “passed” instead of "failed”.
>> 
>> sys.capsicum.capsicum-test: some of the tests were skipped because they weren’t run as root. For example: https://ci.freebsd.org/job/FreeBSD-main-amd64-test/29845/testReport/sys.capsicum/capsicum-test/ . Prior to this change the tests were run both as root and as an unprivileged user - this is “easy to fix” by linking the test and adding the appropriate TEST_METADATA to it (it’s a minor regression).
>> 
>> sys/fs/fusefs/default_permissions:Access.eacces passed in CI... https://ci.freebsd.org/job/FreeBSD-main-amd64-test/29845/testReport/junit/sys.fs.fusefs/default_permissions/Access_eacces/ … along most of the other fusefs tests: a few are skipped, but are explicitly marked “DISABLED”, so their prereqs haven’t been met yet per the test: https://ci.freebsd.org/job/FreeBSD-main-amd64-test/29845/testReport/sys.fs.fusefs/write/AioWrite_DISABLED_aio_write/ (some of the tests skipped in CI are lacking “reasons” for why the tests are disabled).
>> 
>> Are you running the test locally? If so, please check the “skip” message (there are a few prereqs that need to be fulfilled in order to run the tests).
> 
> Yes I'm running them locally.  It turns out the fusefs tests were
> skipping because vfs.usermount is set to 0 there; the message from the
> tests is just "current user is not allowed to mount" in that case.  So
> that's okay.
> 
> The capsicum tests are skipping because they're run with
> required_user="unprivileged" but a few of them require root privileges,
> so there is indeed some regression from commit
> 3fb03fb35b20d259e0a3a07fc48f2ecc3c5da6e8.


Well, some good news and bad news on this front. Good news: the tests will be run as root again as well as unprivileged users (diff pending), restoring test coverage. Bad news is that there will be “test result bloat”: there’s no way to really get around the fact that a ton of tests when run as an unprivileged user will be skipped with the same message (“requires root”). This can be fixed with some light refactoring, but it’s not a simple Makefile change like the (soon-to-be) proposed change will be.

Some of the other permutations previously allowed with capsicum-test available in the larger test suite might need to be added (there are additional arguments which support different behavior in `capsicum-test-main.cc`), but this isn’t a regression since this was the way things were prior to 3fb03fb35b20d259e0a3a07fc48f2ecc3c5da6e8 . The only real net-change in behavior in the tests is that functional.sh explicitly tested root and non-root users and functional.sh ran the entire test suite, unlike kyua does with googletest now (kyua runs each testcase in isolation, process-wise).

-Enji