Re: git: 3e845b109056 - main - ping: fix listing test cases when scapy is not installed

On Sat, May 2, 2026 at 9:24 AM Gleb Smirnoff <glebius@freebsd.org> wrote:
>
> On Sat, May 02, 2026 at 03:20:41PM +0000, Alan Somers wrote:
> A> commit 3e845b1090565912375c5578cf0399d27b7fa70c
> A> Author:     Alan Somers <asomers@FreeBSD.org>
> A> AuthorDate: 2026-05-01 23:06:35 +0000
> A> Commit:     Alan Somers <asomers@FreeBSD.org>
> A> CommitDate: 2026-05-02 15:20:16 +0000
> A>
> A>     ping: fix listing test cases when scapy is not installed
> A>
> A>     The ATF-python test program was attempting to list test cases that
> A>     require scapy.  But it attempted to import the scapy module before the
> A>     test cases had been listed, resulting in an ImportError that kyua
> A>     interpreted as a test program crash.
> A>
> A>     Fix this behavior by handling that ImportError well enough to list test
> A>     cases, but not run them.  If scapy isn't present, Kyua will refuse to
> A>     run the test cases.  But it needs to be able to list them in order to
> A>     know to skip them.
> A>
> A>     Sponsored by:           ConnectWise
> A>     MFC after:              2 weeks
> A>     Reviewed by:            maxim
> A>     Differential Revision:  https://reviews.freebsd.org/D56765
>
> Sorry for not reviewing in time.  I don't think Kyua expects a test program to
> be able to list the test cases, but then fail to run them.
>
> Isn't there a standard way of dealing with the problem?  Many tests have
> @pytest.mark.require_progs(["scapy"]) in them.
>
> --
> Gleb Smirnoff

Yes, these tests already include @pytest.mark.require_progs(["scapy"])
correctly.  That's how Kyua is supposed to learn that it can't run
those test cases.  You can tell by running the program manually,
bypassing kyua:

<after my patch>
$ ./test_ping.py -
Content-Type: application/X-atf-tp; version="1"

ident: TestPing::test_ping[_4_c1_s56_t1_localhost]
descr: Test ping
has.cleanup: true
require.config: unprivileged_user
require.user: root
...
ident: TestPing::test_pinger[_0_0]
descr: Test ping using pinger(), a reply faker
has.cleanup: true
require.user: root
require.progs: scapy
...

Notice that the first test case does not require scapy, but the second
one does.  This is how Kyua learns that it can't execute the second
test if scapy is not installed.  However, prior to my commit, if scapy
was not installed, running test_ping.py -l would fail with exit status
2 and produce no output, and Kyua would interpret that as a broken
test program, like this:

test_ping.py:__test_cases_list__