From nobody Fri May 19 14:48:43 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QN8qX2PNDz4C2RH; Fri, 19 May 2023 14:48:56 +0000 (UTC) (envelope-from arichardson.kde@gmail.com) Received: from mail-il1-f169.google.com (mail-il1-f169.google.com [209.85.166.169]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QN8qX0hHbz3rZ5; Fri, 19 May 2023 14:48:56 +0000 (UTC) (envelope-from arichardson.kde@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-il1-f169.google.com with SMTP id e9e14a558f8ab-3318baede4cso8704325ab.1; Fri, 19 May 2023 07:48:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684507735; x=1687099735; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ric9TSBeFlPFAk30qksqUjCJN8W+YTtXOKuuGaRPYyk=; b=XxTRb0mp5Bu85jcPRbOJm/Jgk+9jP/3Kp7aO42UWwY4vBhbQBYQROv8cDswabz12H/ XkuOnKcVz/1scAJuB7ABNaNmJNPeF6U1FqvXosBjZrFST+SaYXWzXiwPOoVNHTEyRZ5Y Ys8cPDVn+7u/fUEb3jy39soOP47uFdkPo/gk1LMIOdLwQ1DnUnnozTYs1ZWpYC9EDgja ozDscMQje27v+5DIiP5kwMrYaGVod8TW4pM10O4kXXLM2roHb77GTgXL0FYoIWAeC+8l hKO8KAqi8SxyOYPUDecyiZex7F6rz6iqrT0wYgiRHwOQ/njOjzEKyKb/8ufcPPp4Kpkl xeHw== X-Gm-Message-State: AC+VfDxTjYm2tLJ5QdjTCw08XLbTpDnEw0MkOCIU2HC6Bi2yBprVzY5n 5F2iX1Kgq4f0Mb8Nlh/+nw8iT2LAYnQ= X-Google-Smtp-Source: ACHHUZ4AMiHORfyri6KkPDeTFdHGRetwqiwxDg3ZhIw2lPGAycTXPALAzPMRv2gmmxeNMuv1CnJ7TA== X-Received: by 2002:a92:4a0a:0:b0:330:f026:4a8d with SMTP id m10-20020a924a0a000000b00330f0264a8dmr1324182ilf.22.1684507734820; Fri, 19 May 2023 07:48:54 -0700 (PDT) Received: from mail-io1-f44.google.com (mail-io1-f44.google.com. [209.85.166.44]) by smtp.gmail.com with ESMTPSA id q9-20020a0566380d0900b0040f8b6933f0sm1205566jaj.74.2023.05.19.07.48.54 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 19 May 2023 07:48:54 -0700 (PDT) Received: by mail-io1-f44.google.com with SMTP id ca18e2360f4ac-76f06c0f58cso88525639f.1; Fri, 19 May 2023 07:48:54 -0700 (PDT) X-Received: by 2002:a05:6602:29ad:b0:76c:67c3:6bb0 with SMTP id u13-20020a05660229ad00b0076c67c36bb0mr1326151ios.9.1684507734049; Fri, 19 May 2023 07:48:54 -0700 (PDT) List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 References: <202305191349.34JDnp8J060770@gitrepo.freebsd.org> In-Reply-To: From: Alexander Richardson Date: Fri, 19 May 2023 07:48:43 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: git: 40b287054521 - main - mi_startup: Instrument the bubblesort with TSLOG To: Hans Petter Selasky Cc: Colin Percival , src-committers , "" , dev-commits-src-main@freebsd.org Content-Type: multipart/alternative; boundary="0000000000001c0b8c05fc0d059d" X-Rspamd-Queue-Id: 4QN8qX0hHbz3rZ5 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000001c0b8c05fc0d059d Content-Type: text/plain; charset="UTF-8" On Fri, 19 May 2023, 07:07 Hans Petter Selasky, wrote: > On 5/19/23 15:49, Colin Percival wrote: > > The branch main has been updated by cperciva: > > > > URL: > https://cgit.FreeBSD.org/src/commit/?id=40b287054521f0a92e5ae9a26e6a87d17ee85eea > > > > commit 40b287054521f0a92e5ae9a26e6a87d17ee85eea > > Author: Colin Percival > > AuthorDate: 2023-05-19 13:46:42 +0000 > > Commit: Colin Percival > > CommitDate: 2023-05-19 13:46:42 +0000 > > > > mi_startup: Instrument the bubblesort with TSLOG > > > > The bubblesort of SYSINITs is currently responsible for 7% of the > > kernel boot time when booting a 1 CPU / 128 MB VM under Firecracker. > > > > It needs to be replaced with a faster sort, but until that happens > > at least instrumenting it with TSLOG makes it show up in > flamecharts. > > --- > > sys/kern/init_main.c | 2 ++ > > 1 file changed, 2 insertions(+) > > > > diff --git a/sys/kern/init_main.c b/sys/kern/init_main.c > > index 1974c4e68ce4..e4cb501bc57b 100644 > > --- a/sys/kern/init_main.c > > +++ b/sys/kern/init_main.c > > @@ -255,6 +255,7 @@ restart: > > * Perform a bubble sort of the system initialization objects by > > * their subsystem (primary key) and order (secondary key). > > */ > > + TSENTER2("bubblesort"); > > for (sipp = sysinit; sipp < sysinit_end; sipp++) { > > for (xipp = sipp + 1; xipp < sysinit_end; xipp++) { > > if ((*sipp)->subsystem < (*xipp)->subsystem || > > @@ -266,6 +267,7 @@ restart: > > *xipp = save; > > } > > } > > + TSEXIT2("bubblesort"); > > > > last = SI_SUB_COPYRIGHT; > > #if defined(VERBOSE_SYSINIT) > > > > Hi Colin, > > If all kernel modules and the kernel could sort their SYSINIT() and > SYSUNINIT() data at compile time, then all you need to do, is to merge > two sorted lists, when loading new modules. > > Maybe this even could be part of the compiler's existing __constructor > attribute. In FreeBSD we have an example of build boot loader modules, > and statically sorting all sysinit data at compile time. See the tool I > made many years ago for this purpose: > > stand/usb/tools/sysinit.c > > What do you think? > > --HPS > A somewhat minimal improvement can be seen here: https://reviews.freebsd.org/D39916. I noticed the same slowness when booting on qemu with tracing enabled. Sorting at compile time would be ideal and in theory the priority argument in the constructor attribute should work. > --0000000000001c0b8c05fc0d059d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Fri, 19 May 2023, 07:07 Hans Petter Selasky, <hps@selasky.org> wrote:
On 5/19/23 15:49, Colin Percival wrote:
> The branch main has been updated by cperciva:
>
> URL: https://cgit.FreeBSD.org/src/commit/?id=3D40b287054521f0a92e5ae9a26e6a87= d17ee85eea
>
> commit 40b287054521f0a92e5ae9a26e6a87d17ee85eea
> Author:=C2=A0 =C2=A0 =C2=A0Colin Percival <cperciva@FreeBSD.org>=
> AuthorDate: 2023-05-19 13:46:42 +0000
> Commit:=C2=A0 =C2=A0 =C2=A0Colin Percival <cperciva@FreeBSD.org>=
> CommitDate: 2023-05-19 13:46:42 +0000
>
>=C2=A0 =C2=A0 =C2=A0 mi_startup: Instrument the bubblesort with TSLOG >=C2=A0 =C2=A0 =C2=A0
>=C2=A0 =C2=A0 =C2=A0 The bubblesort of SYSINITs is currently responsibl= e for 7% of the
>=C2=A0 =C2=A0 =C2=A0 kernel boot time when booting a 1 CPU / 128 MB VM = under Firecracker.
>=C2=A0 =C2=A0 =C2=A0
>=C2=A0 =C2=A0 =C2=A0 It needs to be replaced with a faster sort, but un= til that happens
>=C2=A0 =C2=A0 =C2=A0 at least instrumenting it with TSLOG makes it show= up in flamecharts.
> ---
>=C2=A0 =C2=A0sys/kern/init_main.c | 2 ++
>=C2=A0 =C2=A01 file changed, 2 insertions(+)
>
> diff --git a/sys/kern/init_main.c b/sys/kern/init_main.c
> index 1974c4e68ce4..e4cb501bc57b 100644
> --- a/sys/kern/init_main.c
> +++ b/sys/kern/init_main.c
> @@ -255,6 +255,7 @@ restart:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 * Perform a bubble sort of the system initi= alization objects by
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 * their subsystem (primary key) and order (= secondary key).
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 */
> +=C2=A0 =C2=A0 =C2=A0TSENTER2("bubblesort");
>=C2=A0 =C2=A0 =C2=A0 =C2=A0for (sipp =3D sysinit; sipp < sysinit_end= ; sipp++) {
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0for (xipp =3D si= pp + 1; xipp < sysinit_end; xipp++) {
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0if ((*sipp)->subsystem < (*xipp)->subsystem ||
> @@ -266,6 +267,7 @@ restart:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0*xipp =3D save;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0}
>=C2=A0 =C2=A0 =C2=A0 =C2=A0}
> +=C2=A0 =C2=A0 =C2=A0TSEXIT2("bubblesort");
>=C2=A0 =C2=A0
>=C2=A0 =C2=A0 =C2=A0 =C2=A0last =3D SI_SUB_COPYRIGHT;
>=C2=A0 =C2=A0#if defined(VERBOSE_SYSINIT)
>

Hi Colin,

If all kernel modules and the kernel could sort their SYSINIT() and
SYSUNINIT() data at compile time, then all you need to do, is to merge
two sorted lists, when loading new modules.

Maybe this even could be part of the compiler's existing __constructor =
attribute. In FreeBSD we have an example of build boot loader modules,
and statically sorting all sysinit data at compile time. See the tool I made many years ago for this purpose:

stand/usb/tools/sysinit.c

What do you think?

--HPS


A somewhat minimal improvement can be seen= here: https://reviews.freeb= sd.org/D39916. I noticed the same slowness when booting on qemu with tr= acing enabled.

Sorting a= t compile time would be ideal and in theory the priority argument in the co= nstructor attribute should work.
--0000000000001c0b8c05fc0d059d--