From nobody Mon Feb 26 16:32:24 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tk5kg63zPz5BkFS for ; Mon, 26 Feb 2024 16:32:43 +0000 (UTC) (envelope-from igor.ostapenko@pm.me) Received: from mail-4322.protonmail.ch (mail-4322.protonmail.ch [185.70.43.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tk5kf5gS1z4TfP; Mon, 26 Feb 2024 16:32:42 +0000 (UTC) (envelope-from igor.ostapenko@pm.me) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=pm.me header.s=protonmail3 header.b=cUxsILeR; dmarc=pass (policy=quarantine) header.from=pm.me; spf=pass (mx1.freebsd.org: domain of igor.ostapenko@pm.me designates 185.70.43.22 as permitted sender) smtp.mailfrom=igor.ostapenko@pm.me DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail3; t=1708965160; x=1709224360; bh=/wKvrCl6f6d3WCEonHU4FFCvXj9/3CSYWH2XNgw563w=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=cUxsILeRBP89mhOu/T51Jg9LP0GZjNSKRDxjeSPzDM3zn3Zxm8O8Ns8DhQPwhRQ0o PSIwCVQBnJ0ObQm2t8fhvp4Zw5iUf4P1zSxT4dE7/iRZDOM+QEJInTv2Uhp+CjFGuv 5HtiUojoebOMvcLogUobdURBDpB1WUAPb77P4H/asHhIn/NugQF43TMOoHBwfIoJW2 o0Z2pmLBic+5GqNAqQlE/DzZa8n/6AjhnNAm4fCCpaFkFPKOGg3of6/LPj/FUOvmZC HieOZGD2KZZ1q0z7Q6jjDkYEvTjO9ughEouHk+L7Gobn9ICVf9SerNYvj5irD4ehd5 d6MvXomQBhVFQ== Date: Mon, 26 Feb 2024 16:32:24 +0000 To: =?utf-8?Q?Olivier_Cochard-Labb=C3=A9?= From: Igor Ostapenko Cc: "freebsd-hackers@freebsd.org" Subject: Re: Add jail execution environment support to the FreeBSD test suite Message-ID: In-Reply-To: References: <2bjQNp1msrv-_AqyamMun6kY-SCqbgPm3Q7DqVQHAYlqvFkiE1i85svfIT-QQdUG1cg3cKippyTyv8Z-5nbLu4WaMutgZQ7KT-YYo_5Pbro=@pm.me> Feedback-ID: 8300135:user:proton List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.20 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[pm.me,quarantine]; R_SPF_ALLOW(-0.20)[+ip4:185.70.43.0/24:c]; R_DKIM_ALLOW(-0.20)[pm.me:s=protonmail3]; RWL_MAILSPIKE_VERYGOOD(-0.20)[185.70.43.22:from]; MIME_GOOD(-0.10)[text/plain]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:62371, ipnet:185.70.43.0/24, country:CH]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; TO_DN_EQ_ADDR_SOME(0.00)[]; DKIM_TRACE(0.00)[pm.me:+] X-Rspamd-Queue-Id: 4Tk5kf5gS1z4TfP On Friday, February 23rd, 2024 at 11:41 AM, Olivier Cochard-Labb=C3=A9 wrote: >=20 > On Thu, Feb 22, 2024 at 9:58=E2=80=AFPM wrote: >=20 > >=20 > > There is a proposal to improve the FreeBSD test suite. > >=20 >=20 >=20 > Thanks a lot for working on improving the parallel tests! >=20 > At work, we are using: > 1. a Nanobsd based, so a lot of WITHOUT_ : WITHOUT_JAIL, WITHOUT_PF, no V= NET (a nightmare to debug network stack with it), etc.; > 2. The standard full test suite (about 8200 tests) is reduced to about 74= 00 tests with this lighter nanobsd; > 3. We have been using kyua parallelism=3D8 for years, and I had to skip o= nly about 10-20 tests that were not compliant with parallel mode. >=20 > It toke about 23 minutes to run them (in a bhyve VM): >=20 > Test cases: 7429 total, 905 skipped, 29 expected failures, 1 broken, 4 fa= iled > Start time: 2024-02-21T00:15:42.527291Z > End time: 2024-02-21T00:38:57.036211Z >=20 > Because we were using WITHOUT_JAIL, to improve the time spent, one of my = ideas was to divide the tests into groups of 4 or 8 and run as many bhyve V= Ms in parallel. > I assumed that by running regression tests in a jail, I was testing the j= ail system as much as the test object (and for our use case, I don=E2=80= =99t care about the jail because we are not using it). > So using the bhyve VM I had the feeling the tests were more confined. >=20 > Now if WITH_JAIL becomes mandatory to be able to run the test, I will jus= t have to modify my build system (like by adding a WITHOUT_JAIL just at ins= tallworld time and not during buildworld). >=20 It would be great if you can test the patch. The general test plan instructions are here: https://reviews.freebsd.org/D4= 2350. It includes instructions for your case WITHOUT_JAIL. It's expected that you need no changes in your build system, the Kyua should be built without jail support and your usual test runs should work as usually. And any execenv=3D= jail based test on your way should be automatically skipped by such instance of = Kyua. Best regards, Igor.