From nobody Tue Feb 27 18:32:31 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkmLj323Jz5CCTj for ; Tue, 27 Feb 2024 18:32:45 +0000 (UTC) (envelope-from igor.ostapenko@pm.me) Received: from mail-4316.protonmail.ch (mail-4316.protonmail.ch [185.70.43.16]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkmLh0rJ3z4y0G for ; Tue, 27 Feb 2024 18:32:44 +0000 (UTC) (envelope-from igor.ostapenko@pm.me) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=pm.me header.s=protonmail3 header.b=AEMKYrXw; dmarc=pass (policy=quarantine) header.from=pm.me; spf=pass (mx1.freebsd.org: domain of igor.ostapenko@pm.me designates 185.70.43.16 as permitted sender) smtp.mailfrom=igor.ostapenko@pm.me DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail3; t=1709058761; x=1709317961; bh=p/bc+sBW7mNpxenSh78qXiK9GBlUeJNqqCiUGmku/kE=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=AEMKYrXwElRHgIg6ki4Bgd0buGifPFQQrDEPnoLx7LyyaKwwJj4mvEVuJzqLwsVaw L35mjyDWQGoUePDjQXhnRNLU6EQpgaTxkfLwMASnOYrzTUxStyJDLqUSJybgKU28Ve 7TQ69SLHxIqVqLuAXfyAmHfRn2KKeyBs5TGRfw+cF6Vmf6KOWkIDukx4O+iQTaf1O5 z4RmvcbzntXQk85xXNXEpORWYO5blOTfhtrwi5RxbMYgB/8EjySI9id59Xnhruub3u stwPrmWo85+9fXf9btlrOvYhfP5SXbWbsc0Fkb4CHNWe4+P6JVKFhrb9XPsOMcmbZR OYbUd8AqkQTSg== Date: Tue, 27 Feb 2024 18:32:31 +0000 To: "freebsd-hackers@freebsd.org" From: Igor Ostapenko Subject: Re: Add jail execution environment support to the FreeBSD test suite Message-ID: In-Reply-To: <2bjQNp1msrv-_AqyamMun6kY-SCqbgPm3Q7DqVQHAYlqvFkiE1i85svfIT-QQdUG1cg3cKippyTyv8Z-5nbLu4WaMutgZQ7KT-YYo_5Pbro=@pm.me> References: <2bjQNp1msrv-_AqyamMun6kY-SCqbgPm3Q7DqVQHAYlqvFkiE1i85svfIT-QQdUG1cg3cKippyTyv8Z-5nbLu4WaMutgZQ7KT-YYo_5Pbro=@pm.me> Feedback-ID: 8300135:user:proton List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.20 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[pm.me,quarantine]; R_SPF_ALLOW(-0.20)[+ip4:185.70.43.0/24]; R_DKIM_ALLOW(-0.20)[pm.me:s=protonmail3]; RWL_MAILSPIKE_VERYGOOD(-0.20)[185.70.43.16:from]; MIME_GOOD(-0.10)[text/plain]; ASN(0.00)[asn:62371, ipnet:185.70.43.0/24, country:CH]; RCPT_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_DN_EQ_ADDR_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[pm.me:+] X-Rspamd-Queue-Id: 4TkmLh0rJ3z4y0G Hi, The patch was updated after the recent discussion. Currently, the patch provides the following new functionality, from bottom = to top: 1 ATF based tests - The new "execenv" metadata property can be set to explicitly ask for an execution environment: "host" or "jail". If it's not defined, as all existing tests do, then it implicitly means "host". - The new "execenv.jail" metadata property can be optionally defined to ask Kyua to use specific jail(8) parameters during creation of a temporary jail. An example is "vnet allow.raw_sockets". 2 Kyuafile - The same new metadata properties can be defined on Kyuafile level: "execenv" and "execenv_jail". - Note that historically ATF uses dotted style of metadata naming, while Kyua uses underscore style. Hence "execenv.jail" vs. "execenv_jail". 3 kyua.conf, kyua CLI - The new "execenv" engine configuration variable can be set to a list of execution environments to run only tests designed for. Tests of not liste= d environments are skipped. - By default, this variable lists all execution environments supported by a Kyua binary, e.g. execenv=3D"host jail". - This variable can be changed via "kyua.conf" or via kyua CLI's "-v" parameter. For example, "kyua -v execenv=3Dhost test" will run only host-based tests and skip jail-based ones. - Current value of this variable can be examined with "kyua config". The patch is https://reviews.freebsd.org/D42350. Any help with review and testing is welcome. Its test plan covers the details and refers to the demo patch of how existing tests could be converted to be run in a jail. Best regards, Igor. On Thursday, February 22nd, 2024 at 10:57 PM, igor.ostapenko@pm.me wrote: >=20 >=20 > Hi FreeBSD developers, >=20 > There is a proposal to improve the FreeBSD test suite. >=20 >=20 > 1 The Problem >=20 > The FreeBSD test suite is based on the Kyua framework. The latter support= s > running tests in parallel. However, some tests cannot be run in parallel = and > are marked with is_exclusive=3D"true" metadata, which makes Kyua run such= tests > in sequence. >=20 > Many tests are not meant to be exclusive conceptually, they are so for ve= ry > simple technical reasons. For instance, some network related tests are ba= sed > on jail and vnet usage. It's convenient for such tests and it provides a = lot > of isolation already not to conflict with other tests. But they are still > marked as exclusive due to the shared space of jail names, routing, etc. >=20 > The project seeks more tests, and it's kind of a trend for new tests like > jail/vnet based ones to be created as is_exclusive=3D"true" from the very > beginning. It only piles up the suite with exclusive tests, e.g. new test= s > from my side faced a fair question from a reviewer whether they could be > re-designed for a parallel run. [1] >=20 > If such tests were 100% isolated they would be able to run in parallel an= d > decrease the test time for CI runs and for the runs within the developmen= t > process. >=20 > And the problem is that trying to add more isolation by a test itself loo= ks to > be a doable task from a glance, but it would add a lot of complexity to a= test > code, or could be found as an impossible task in a specific case. >=20 >=20 > 2 The Idea >=20 > The idea is not new. A test could be running in a jail -- it provides the > required isolation with minimum or zero effort from a test. >=20 >=20 > 3 The Implementation >=20 > There is a lot of work done already and the working patch passed the init= ial > review (thanks to markj@ and ngie@). [2] >=20 > It adds a new concept to the Kyua framework -- an execution environment. = Two > new metadata were added for that: execenv and execenv_jail. >=20 > execenv is a switch to select an environment. If a test's metadata define= s > execenv=3D"jail" then Kyua will create a temporary jail, run such test wi= thin > it, and remove the jail. If execenv=3D"host" is provided or execenv metad= ata is > undefined then Kyua will run such test as it does today. >=20 > execenv_jail metadata takes effect only in case of execenv=3D"jail". It a= llows a > test to request specific parameters for its jail. These parameters are si= mply > arguments to jail(8), e.g. execenv_jail=3D"vnet allow.raw_sockets". >=20 >=20 > 4 The Adoption >=20 > ATF based tests can easily define this new metadata via Kyuafile or direc= tly, > e.g. for atf-sh based tests: >=20 > test_head() > { > atf_set descr "Test foo in case of bar" > atf_set require.user root > atf_set execenv jail > atf_set execenv.jail vnet allow.raw_sockets > } >=20 > Non-ATF based ones will do it via Kyuafile. Our test suite does it throug= h a > Makefile: >=20 > TEST_METADATA+=3D execenv=3D"jail" > TEST_METADATA+=3D execenv_jail=3D"vnet allow.raw_sockets" >=20 > The patch got some little evolution, I started with a single execenv_jail > metadata, and during the patch discussion and review, I ended up with two > knobs: execenv and execenv_jail. It turned out to be a cleaner and less t= ricky > interface such way. The evolution reasoning can be found in the history o= f the > respective Differential. [2] >=20 >=20 > 5 MFC Concerns >=20 > For now, I see at least one issue from the usual project workflow perspec= tive. > Let's imagine that the Kyua framework got this execenv feature committed = to > 15-CURRENT, we started to convert existing tests and create new ones to u= se > execenv=3D"jail". If some feature or a bug fix needs to be ported back to > 14-STABLE or 13-STABLE, then "old" Kyua without execenv feature will fail= to > run such tests: >=20 > kyua: E: Load of 'Kyuafile' failed: Failed to load Lua file 'Kyuafile': K= yuafile:9: Unknown metadata property execenv. >=20 > From a combinatorics perspective, the first three options pop up to deal = with > that: > a) Patch Kyua the same way for the supported STABLE branches so it will b= e > able to run back ported tests based on execenv=3D"jail" (it's not system = ABI > change after all) > b) Exclusively patch Kyua framework for the supported STABLE branches to > simply skip such tests (does not look to provide much benefit) > c) Do not back port tests, only the fix/feature itself (kind of a bad ide= a) >=20 >=20 > 6 The Demo >=20 > My test environment showed promising run time numbers for almost the whol= e > test suite (ZFS excluded). One of the tests yielded 36 min with test > parallelism improvement versus 1 h 25 min without. In my case with 8 core= s, > the suite runs about 2 times faster with the improvement. [3] >=20 >=20 > 7 Action Points >=20 > My current vision of the plan looks as follows: > - [ ] community: Review, testing, comments -- probably we want to change = the > design > - [ ] committers: Help with the main commit -- it should hit freebsd/kyua > GitHub fork first [4], then vendor branch, and merge to > main after > - [ ] igoro: Provide the subsequent PRs to separate FreeBSD specifics and= fix > existing Kyua tests > - [ ] igoro: Provide the PRs to add brand new tests of Kyua itself to cov= er > the new feature > - [ ] igoro: Provide the respective documentation updates > - [ ] igoro: Migrate some of the existing tests for the start, e.g. netpf= il/pf > - [ ] committers: Help with review and respective commits/merges >=20 > The plan is not strict, it depends on the discussion and interest of > volunteers. >=20 > I hope that this proposal is found valuable for the project. If so, any h= elp > is appreciated. >=20 >=20 > [1] New tests exclusivity concern: https://reviews.freebsd.org/D42314 > [2] The Kyua patch: https://reviews.freebsd.org/D42350 > [3] The whole test suite demo: https://reviews.freebsd.org/D42410 > [4] The respective PR to the fork: https://github.com/freebsd/kyua/pull/2= 24 >=20 >=20 > Best regards, Igor.