svn commit: r366708 - in head/sbin/pfctl/tests: . files
Alex Richardson
arichardson at FreeBSD.org
Wed Oct 14 17:39:52 UTC 2020
Author: arichardson
Date: Wed Oct 14 17:39:50 2020
New Revision: 366708
URL: https://svnweb.freebsd.org/changeset/base/366708
Log:
Rewrite pfctl_test in C to reduce testsuite run time
The new C test takes 25 seconds on QEMU-RISC-V, wheras the shell version
takes 332 seconds.
Even with the latest optimizations to atf-sh this test still takes a few
seconds to startup in QEMU. Re-writing it in C reduces the runtime for a
single test from about 2-3 seconds to less than .5 seconds. Since there
are ~80 tests, this adds up to about 3-4 minutes.
This may not seem like a big speedup, but before the recent optimizations
to avoid atf_get_srcdir, each test took almost 100 seconds on QEMU RISC-V
instead of 3. This also significantly reduces the time it takes to list
the available test cases, which speeds up running the tests via kyua:
```
root at qemu-riscv64-alex:~ # /usr/bin/time kyua test -k /usr/tests/sbin/pfctl/Kyuafile pfctl_test_old
...
158/158 passed (0 failed)
332.08 real 42.58 user 286.17 sys
root at qemu-riscv64-alex:~ # /usr/bin/time kyua test -k /usr/tests/sbin/pfctl/Kyuafile pfctl_test
158/158 passed (0 failed)
24.96 real 9.75 user 14.26 sys
root at qemu-riscv64-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test pf1001
pfctl_test: WARNING: Running test cases outside of kyua(1) is unsupported
pfctl_test: WARNING: No isolation nor timeout control is being applied; you may get unexpected failures; see atf-test-case(4)
Running pfctl -o none -nvf /usr/tests/sbin/pfctl/./files/pf1001.in
---
binat on em0 inet6 from fc00::/64 to any -> fc00:0:0:1::/64
binat on em0 inet6 from any to fc00:0:0:1::/64 -> fc00::/64
---
passed
0.17 real 0.06 user 0.08 sys
root at qemu-riscv64-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test_old pf1001
pfctl_test_old: WARNING: Running test cases outside of kyua(1) is unsupported
pfctl_test_old: WARNING: No isolation nor timeout control is being applied; you may get unexpected failures; see atf-test-case(4)
Id Refs Name
141 1 pf
Executing command [ pfctl -o none -nvf - ]
passed
1.73 real 0.25 user 1.41 sys
root at qemu-riscv64-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test_old -l > /dev/null
24.36 real 2.26 user 21.86 sys
root at qemu-riscv64-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test -l > /dev/null
0.04 real 0.02 user 0.01 sys
```
The speedups are even more noticeable on CHERI-RISC-V (since QEMU runs
slower when emulating CHERI instructions):
```
root at qemu-cheri-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test_new -l > /dev/null
0.51 real 0.49 user 0.00 sys
root at qemu-cheri-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test -l > /dev/null
34.20 real 32.69 user 0.16 sys
root at qemu-cheri-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test pf1001
pfctl_test: WARNING: Running test cases outside of kyua(1) is unsupported
pfctl_test: WARNING: No isolation nor timeout control is being applied; you may get unexpected failures; see atf-test-case(4)
Id Refs Name
147 1 pf
Executing command [ pfctl -o none -nvf - ]
passed
5.74 real 5.41 user 0.03 sys
root at qemu-cheri-alex:/usr/tests/sbin/pfctl # /usr/bin/time ./pfctl_test_new pf1001
pfctl_test_new: WARNING: Running test cases outside of kyua(1) is unsupported
pfctl_test_new: WARNING: No isolation nor timeout control is being applied; you may get unexpected failures; see atf-test-case(4)
Running pfctl -o none -nvf /usr/tests/sbin/pfctl/./files/pf1001.in
---
binat on em0 inet6 from fc00::/64 to any -> fc00:0:0:1::/64
binat on em0 inet6 from any to fc00:0:0:1::/64 -> fc00::/64
---
passed
0.68 real 0.66 user 0.00 sys
root at qemu-cheri-alex:/usr/tests/sbin/pfctl #
```
Reviewed By: kp
Differential Revision: https://reviews.freebsd.org/D26779
Added:
head/sbin/pfctl/tests/pfctl_test.c (contents, props changed)
head/sbin/pfctl/tests/pfctl_test_list.inc (contents, props changed)
Deleted:
head/sbin/pfctl/tests/files/pfctl_test_descr.sh
head/sbin/pfctl/tests/pfctl_test.sh
Modified:
head/sbin/pfctl/tests/Makefile
head/sbin/pfctl/tests/files/Makefile
Modified: head/sbin/pfctl/tests/Makefile
==============================================================================
--- head/sbin/pfctl/tests/Makefile Wed Oct 14 15:50:28 2020 (r366707)
+++ head/sbin/pfctl/tests/Makefile Wed Oct 14 17:39:50 2020 (r366708)
@@ -2,9 +2,11 @@
PACKAGE= tests
-ATF_TESTS_SH= pfctl_test \
- macro
+ATF_TESTS_C= pfctl_test
+ATF_TESTS_SH= macro
+LIBADD+= sbuf
SUBDIR+= files
+WARNS=6
.include <bsd.test.mk>
Modified: head/sbin/pfctl/tests/files/Makefile
==============================================================================
--- head/sbin/pfctl/tests/files/Makefile Wed Oct 14 15:50:28 2020 (r366707)
+++ head/sbin/pfctl/tests/files/Makefile Wed Oct 14 17:39:50 2020 (r366708)
@@ -7,6 +7,5 @@ BINDIR= ${TESTSDIR}
# We use ${.CURDIR} as workaround so that the glob patterns work.
FILES!= echo ${.CURDIR}/pf????.in ${.CURDIR}/pf????.include ${.CURDIR}/pf????.ok
-FILES+= ${.CURDIR}/pfctl_test_descr.sh
.include <bsd.progs.mk>
Added: head/sbin/pfctl/tests/pfctl_test.c
==============================================================================
--- /dev/null 00:00:00 1970 (empty, because file is newly added)
+++ head/sbin/pfctl/tests/pfctl_test.c Wed Oct 14 17:39:50 2020 (r366708)
@@ -0,0 +1,230 @@
+/*-
+ * SPDX-License-Identifier: BSD-2-Clause
+ *
+ * Copyright 2020 Alex Richardson <arichardson at FreeBSD.org>
+ *
+ * This software was developed by SRI International and the University of
+ * Cambridge Computer Laboratory (Department of Computer Science and
+ * Technology) under DARPA contract HR0011-18-C-0016 ("ECATS"), as part of the
+ * DARPA SSITH research programme.
+ *
+ * This work was supported by Innovate UK project 105694, "Digital Security by
+ * Design (DSbD) Technology Platform Prototype".
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ * 1. Redistributions of source code must retain the above copyright notice,
+ * this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright notice,
+ * this list of conditions and the following disclaimer in the documentation
+ * and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+ * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+ * DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY
+ * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+ * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+#include <sys/cdefs.h>
+__FBSDID("$FreeBSD$");
+
+#include <sys/types.h>
+#include <sys/param.h>
+#include <err.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <spawn.h>
+#include <sys/module.h>
+#include <sys/sbuf.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+
+#include <atf-c.h>
+
+/*
+ * Tests 0001-0999 are copied from OpenBSD's regress/sbin/pfctl.
+ * Tests 1001-1999 are ours (FreeBSD's own).
+ *
+ * pf: Run pfctl -nv on pfNNNN.in and check that the output matches pfNNNN.ok.
+ * Copied from OpenBSD. Main differences are some things not working
+ * in FreeBSD:
+ * * The action 'match'
+ * * The command 'set reassemble'
+ * * The 'from'/'to' options together with 'route-to'
+ * * The option 'scrub' (it is an action in FreeBSD)
+ * * Accepting undefined routing tables in actions (??: see pf0093.in)
+ * * The 'route' option
+ * * The 'set queue def' option
+ * selfpf: Feed pfctl output through pfctl again and verify it stays the same.
+ * Copied from OpenBSD.
+ */
+
+static bool
+check_pf_module_available()
+{
+ int modid;
+ struct module_stat stat;
+
+ if ((modid = modfind("pf")) < 0) {
+ warn("pf module not found");
+ return false;
+ }
+ stat.version = sizeof(struct module_stat);
+ if (modstat(modid, &stat) < 0) {
+ warn("can't stat pf module id %d", modid);
+ return false;
+ }
+ return (true);
+}
+
+extern char **environ;
+
+static struct sbuf *
+read_fd(int fd, size_t sizehint)
+{
+ struct sbuf *sb;
+ ssize_t count;
+ char buffer[MAXBSIZE];
+
+ sb = sbuf_new(NULL, NULL, sizehint, SBUF_AUTOEXTEND);
+ errno = 0;
+ while ((count = read(fd, buffer, sizeof(buffer) - 1)) > 0) {
+ sbuf_bcat(sb, buffer, count);
+ }
+ ATF_REQUIRE_ERRNO(0, count == 0 && "Should have reached EOF");
+ sbuf_finish(sb); /* Ensure NULL-termination */
+ return (sb);
+}
+
+static struct sbuf *
+read_file(const char *filename)
+{
+ struct stat s;
+ struct sbuf *result;
+ int fd;
+
+ errno = 0;
+ ATF_REQUIRE_EQ_MSG(stat(filename, &s), 0, "cannot stat %s", filename);
+ fd = open(filename, O_RDONLY);
+ ATF_REQUIRE_ERRNO(0, fd > 0);
+ result = read_fd(fd, s.st_size);
+ ATF_REQUIRE_ERRNO(0, close(fd) == 0);
+ return (result);
+}
+
+static void
+run_pfctl_test(const char *input_path, const char *expected_path,
+ const atf_tc_t *tc)
+{
+ int status;
+ pid_t pid;
+ int pipefds[2];
+ char input_files_path[PATH_MAX];
+ struct sbuf *expected_output;
+ struct sbuf *real_output;
+ posix_spawn_file_actions_t action;
+
+ if (!check_pf_module_available())
+ atf_tc_skip("pf(4) is not loaded");
+
+ /* The test inputs need to be able to use relative includes. */
+ snprintf(input_files_path, sizeof(input_files_path), "%s/files",
+ atf_tc_get_config_var(tc, "srcdir"));
+ ATF_REQUIRE_ERRNO(0, chdir(input_files_path) == 0);
+
+ ATF_REQUIRE_ERRNO(0, pipe(pipefds) == 0);
+ expected_output = read_file(expected_path);
+
+ posix_spawn_file_actions_init(&action);
+ posix_spawn_file_actions_addclose(&action, STDIN_FILENO);
+ posix_spawn_file_actions_addclose(&action, pipefds[1]);
+ posix_spawn_file_actions_adddup2(&action, pipefds[0], STDOUT_FILENO);
+ posix_spawn_file_actions_adddup2(&action, pipefds[0], STDERR_FILENO);
+
+ const char *argv[] = { "pfctl", "-o", "none", "-nvf", input_path,
+ NULL };
+ printf("Running %s %s %s %s %s\n", argv[0], argv[1], argv[2], argv[3],
+ argv[4]);
+ status = posix_spawnp(
+ &pid, "pfctl", &action, NULL, __DECONST(char **, argv), environ);
+ ATF_REQUIRE_EQ_MSG(
+ status, 0, "posix_spawn failed: %s", strerror(errno));
+ posix_spawn_file_actions_destroy(&action);
+ close(pipefds[0]);
+
+ real_output = read_fd(pipefds[1], 0);
+ printf("---\n%s---\n", sbuf_data(real_output));
+ ATF_REQUIRE_EQ(waitpid(pid, &status, 0), pid);
+ ATF_REQUIRE_MSG(WIFEXITED(status),
+ "pfctl returned non-zero! Output:\n %s", sbuf_data(real_output));
+
+ ATF_CHECK_STREQ(sbuf_data(expected_output), sbuf_data(real_output));
+ sbuf_delete(expected_output);
+ sbuf_delete(real_output);
+ close(pipefds[1]);
+}
+
+static void
+do_pf_test(const char *number, const atf_tc_t *tc)
+{
+ char *input_path;
+ char *expected_path;
+ asprintf(&input_path, "%s/files/pf%s.in",
+ atf_tc_get_config_var(tc, "srcdir"), number);
+ asprintf(&expected_path, "%s/files/pf%s.ok",
+ atf_tc_get_config_var(tc, "srcdir"), number);
+ run_pfctl_test(input_path, expected_path, tc);
+ free(input_path);
+ free(expected_path);
+}
+
+static void
+do_selfpf_test(const char *number, const atf_tc_t *tc)
+{
+ char *expected_path;
+ asprintf(&expected_path, "%s/files/pf%s.ok",
+ atf_tc_get_config_var(tc, "srcdir"), number);
+ run_pfctl_test(expected_path, expected_path, tc);
+ free(expected_path);
+}
+
+#define PFCTL_TEST(number, descr) \
+ ATF_TC(pf##number); \
+ ATF_TC_HEAD(pf##number, tc) \
+ { \
+ atf_tc_set_md_var(tc, "descr", descr); \
+ } \
+ ATF_TC_BODY(pf##number, tc) \
+ { \
+ do_pf_test(#number, tc); \
+ } \
+ ATF_TC(selfpf##number); \
+ ATF_TC_HEAD(selfpf##number, tc) \
+ { \
+ atf_tc_set_md_var(tc, "descr", "Self " descr); \
+ } \
+ ATF_TC_BODY(selfpf##number, tc) \
+ { \
+ do_selfpf_test(#number, tc); \
+ }
+#include "pfctl_test_list.inc"
+#undef PFCTL_TEST
+
+ATF_TP_ADD_TCS(tp)
+{
+#define PFCTL_TEST(number, descr) \
+ ATF_TP_ADD_TC(tp, pf##number); \
+ ATF_TP_ADD_TC(tp, selfpf##number);
+#include "pfctl_test_list.inc"
+#undef PFCTL_TEST
+
+ return atf_no_error();
+}
Added: head/sbin/pfctl/tests/pfctl_test_list.inc
==============================================================================
--- /dev/null 00:00:00 1970 (empty, because file is newly added)
+++ head/sbin/pfctl/tests/pfctl_test_list.inc Wed Oct 14 17:39:50 2020 (r366708)
@@ -0,0 +1,118 @@
+/*-
+ * SPDX-License-Identifier: BSD-2-Clause
+ *
+ * Copyright 2020 Alex Richardson <arichardson at FreeBSD.org>
+ *
+ * This software was developed by SRI International and the University of
+ * Cambridge Computer Laboratory (Department of Computer Science and
+ * Technology) under DARPA contract HR0011-18-C-0016 ("ECATS"), as part of the
+ * DARPA SSITH research programme.
+ *
+ * This work was supported by Innovate UK project 105694, "Digital Security by
+ * Design (DSbD) Technology Platform Prototype".
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ * 1. Redistributions of source code must retain the above copyright notice,
+ * this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright notice,
+ * this list of conditions and the following disclaimer in the documentation
+ * and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+ * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+ * DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY
+ * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+ * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * $FreeBSD$
+ */
+
+/*
+ * No include guards since this file is included multiple times by pfctl_test
+ * to avoid duplicating code.
+ */
+PFCTL_TEST(0001, "Pass with labels")
+PFCTL_TEST(0002, "Block/pass")
+PFCTL_TEST(0003, "Block/pass with flags")
+PFCTL_TEST(0004, "Block")
+PFCTL_TEST(0005, "Block with variables")
+PFCTL_TEST(0006, "Variables")
+PFCTL_TEST(0007, "Block/pass with return")
+PFCTL_TEST(0008, "Block with address list")
+PFCTL_TEST(0009, "Block with interface list")
+PFCTL_TEST(0010, "Block/pass with return")
+PFCTL_TEST(0011, "Block/pass ICMP")
+PFCTL_TEST(0012, "Pass to subnets")
+PFCTL_TEST(0013, "Pass quick")
+PFCTL_TEST(0014, "Pass quick IPv6")
+PFCTL_TEST(0016, "Pass with no state")
+PFCTL_TEST(0018, "Address lists")
+PFCTL_TEST(0019, "Lists")
+PFCTL_TEST(0020, "Lists")
+PFCTL_TEST(0022, "Set options")
+PFCTL_TEST(0023, "Block on negated interface")
+PFCTL_TEST(0024, "Variable concatenation")
+PFCTL_TEST(0025, "Antispoof")
+PFCTL_TEST(0026, "Block from negated interface")
+PFCTL_TEST(0028, "Block with log and quick")
+PFCTL_TEST(0030, "Line continuation")
+PFCTL_TEST(0031, "Block policy")
+PFCTL_TEST(0032, "Pass to any")
+PFCTL_TEST(0034, "Pass with probability")
+PFCTL_TEST(0035, "Matching on TOS")
+PFCTL_TEST(0038, "Pass with user")
+PFCTL_TEST(0039, "Ordered opts")
+PFCTL_TEST(0040, "Block/pass")
+PFCTL_TEST(0041, "Anchors")
+PFCTL_TEST(0047, "Pass with labels")
+PFCTL_TEST(0048, "Tables")
+PFCTL_TEST(0049, "Broadcast and network modifiers")
+PFCTL_TEST(0050, "Double macro set")
+PFCTL_TEST(0052, "Set optimization")
+PFCTL_TEST(0053, "Pass with labels")
+PFCTL_TEST(0055, "Set options")
+PFCTL_TEST(0056, "State opts")
+PFCTL_TEST(0057, "Variables")
+PFCTL_TEST(0060, "Pass from multicast")
+PFCTL_TEST(0061, "Dynaddr with netmask")
+PFCTL_TEST(0065, "Antispoof with labels")
+PFCTL_TEST(0067, "Tags")
+PFCTL_TEST(0069, "Tags")
+PFCTL_TEST(0070, "Tags")
+PFCTL_TEST(0071, "Tags")
+PFCTL_TEST(0072, "Tags")
+PFCTL_TEST(0074, "Synproxy")
+PFCTL_TEST(0075, "Block quick with tags")
+PFCTL_TEST(0077, "Dynaddr with netmask")
+PFCTL_TEST(0078, "Table with label")
+PFCTL_TEST(0079, "No-route with label")
+PFCTL_TEST(0081, "Address list and table list with no-route")
+PFCTL_TEST(0082, "Pass with interface, table and no-route")
+PFCTL_TEST(0084, "Source track")
+PFCTL_TEST(0085, "Tag macro expansion")
+PFCTL_TEST(0087, "Optimization rule reordering")
+PFCTL_TEST(0088, "Optimization duplicate rules handling")
+PFCTL_TEST(0089, "TCP connection tracking")
+PFCTL_TEST(0090, "Log opts")
+PFCTL_TEST(0091, "Nested anchors")
+PFCTL_TEST(0092, "Comments")
+PFCTL_TEST(0094, "Address ranges")
+PFCTL_TEST(0095, "Include")
+PFCTL_TEST(0096, "Variables")
+PFCTL_TEST(0097, "Divert-to")
+PFCTL_TEST(0098, "Pass")
+PFCTL_TEST(0100, "Anchor with multiple path components")
+PFCTL_TEST(0101, "Prio")
+PFCTL_TEST(0102, "Address lists with mixed address family")
+PFCTL_TEST(0104, "Divert-to with localhost")
+PFCTL_TEST(1001, "Binat")
+PFCTL_TEST(1002, "Set timeout interval")
+PFCTL_TEST(1003, "ALTQ")
+PFCTL_TEST(1004, "ALTQ with Codel")
+PFCTL_TEST(1005, "PR 231323")
More information about the svn-src-head
mailing list