[Bug 289146] sbin/ipfw/test_add_rule.py:TestAddRule::test_add_action tests failing in CI

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289146

            Bug ID: 289146
           Summary: sbin/ipfw/test_add_rule.py:TestAddRule::test_add_actio
                    n tests failing in CI
           Product: Base System
           Version: CURRENT
          Hardware: Any
                OS: Any
            Status: New
          Keywords: regression
          Severity: Affects Only Me
          Priority: ---
         Component: tests
          Assignee: melifaro@FreeBSD.org
          Reporter: me@svmhdvn.name
                CC: testing@FreeBSD.org

Created attachment 263289
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=263289&action=edit
kyua test report

After
https://cgit.freebsd.org/src/commit/?id=d879f1c8961d99bdbe6727004bfc6c23924526f8
added the HAS_TESTS= variable to sbin/ipfw/Makefile (to reveal the hidden
testcases that were never being run), the following parameterized testcases
inside the 'sbin/ipfw/test_add_rule.py:TestAddRule::test_add_action' testsuite
are now showing up to fail in CI.

sbin/ipfw/test_add_rule.py:TestAddRule::test_add_action[call_420]  ->  failed:
/usr/tests/atf_python/sys/netpfil/ipfw/insns.py:105: ValueError  [0.231s] 
sbin/ipfw/test_add_rule.py:TestAddRule::test_add_action[return]  ->  failed:
/usr/tests/atf_python/sys/netpfil/ipfw/insns.py:105: ValueError  [0.272s] 
sbin/ipfw/test_add_rule.py:TestAddRule::test_add_action[skipto_42]  ->  failed:
/usr/tests/atf_python/sys/netpfil/ipfw/insns.py:105: ValueError  [0.278s] 
sbin/ipfw/test_add_rule.py:TestAddRule::test_add_rule[test_check_state]  -> 
failed: /usr/tests/atf_python/sys/netpfil/ipfw/insns.py:105: ValueError 
[0.223s] 
sbin/ipfw/test_add_rule.py:TestAddRule::test_add_rule[test_eaction_ntp]  -> 
failed: /usr/tests/sbin/ipfw/test_add_rule.py:119: AssertionError  [0.189s] 
sbin/ipfw/test_add_rule.py:TestAddRule::test_add_rule[test_eaction_tcp-setmss] 
->  failed: /usr/tests/sbin/ipfw/test_add_rule.py:119: AssertionError  [0.192s] 
sbin/ipfw/test_add_rule.py:TestAddRule::test_add_rule[test_keep_state]  -> 
failed: /usr/tests/atf_python/sys/netpfil/ipfw/insns.py:105: ValueError 
[0.222s] 
sbin/ipfw/test_add_rule.py:TestAddRule::test_add_rule[test_record_state]  -> 
failed: /usr/tests/atf_python/sys/netpfil/ipfw/insns.py:105: ValueError 
[0.218s] 
sbin/ipfw/test_add_rule.py:TestAddRule::test_add_rule[test_tables]  ->  failed:
/usr/tests/sbin/ipfw/test_add_rule.py:119: AssertionError  [0.202s]

I've attached the full Kyua test report from my test run in this bug, which
contains the stderr from these failing cases.

I have reproduced this error with two methods and attached a sample
test-report.txt:

1. using src in-tree local CI:

# make -C tests/ci CITYPE=full PARALLEL_JOBS=2 VM_MEM_SIZE=4g
KYUA_TEST_FILTERS='sbin/ipfw/test_add_rule.py'

2. using bricoler[0]:

$ bricoler run freebsd-src-regression-suite --param freebsd-src:url=/usr/src
--param freebsd-src:branch= --param
freebsd-src-regression-suite:hypervisor=bhyve --param
freebsd-src-regression-suite:memory=4096 --param
freebsd-src-regression-suite:ncpus=2 --param
freebsd-src-regression-suite:parallelism=1 --param
freebsd-src-regression-suite:tests='sbin/ipfw/test_add_rule.py'

@melifaro I see that you wrote these test cases. Please triage as necessary.

[0] https://github.com/markjdb/bricoler

-- 
You are receiving this mail because:
You are on the CC list for the bug.