From nobody Sun Dec 07 10:30:41 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dPLxJ4SDNz6KCjP for ; Sun, 07 Dec 2025 10:31:00 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4dPLxJ06M7z46sD for ; Sun, 07 Dec 2025 10:31:00 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x52b.google.com with SMTP id 4fb4d7f45d1cf-640ca678745so6121220a12.2 for ; Sun, 07 Dec 2025 02:30:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1765103454; x=1765708254; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=/RQL/2HZWKD9FOcedhtdg31DuuTxKZ3vSUfjLvigzD4=; b=Ymu/zDFZPDXerXNF7OZQRQF336RCCxlTnJTh2rqn08YCbi1pgJGk/Zu7qBqU+9+qUW qq0LOOqyOFo3Gy5VKo18v60wOWwgSU8v5sY6dzxqCohdSZR/oh+TqkSPfp2MU/RLFqgW iLQGLxRwWTJzJ6zzVK7F7U96PQFcwhr76+O5ocIhvUpXYtcU2hAX+BtONjsUnhzFmI0b XMuCk14gf8PWAgRY5LpD2iS01jrveSI82GcluRS2cFV4iV+gCb6QB/dvq2u6WkPSWSEj 8G15BQmgIVMGTvT0qxPjHHRwlJfeyfpUFz6YoiGfsjV4b+RMt0IUpv7xsiTfpM1b5Zsa uylw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765103454; x=1765708254; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=/RQL/2HZWKD9FOcedhtdg31DuuTxKZ3vSUfjLvigzD4=; b=oARaIBR87gYKXUwNb2x1vcx5Gv2tZhDUtXsBKeC0XOkS7n8BkqETb9DIGcRl1WGxU9 uVbQQmqy9uTOOdX4UuhHa7HWJeurL41P3sn+uoig4uZ/Zq47sQA0i+hGmx3PWwNL35Ou y507RNtJuMGWuouvZnWbPReSTFFAeA9SAAbEpCSaJMcRJFHnblr7tChf08bD6LOyBlhC OSYH/+uqwTdctohROmGeTkh0polIC2Hb6sTr7WIteshXTRDSWBe6GrBge7WJ8ASz8bHU GuUWGk9qSYh3SeYFs1IgJYf84aV2emt8WiaBUJx4kjHVLvHJ9xUyjQWhqvFSKvDRjVaS E2Xw== X-Forwarded-Encrypted: i=1; AJvYcCXIU4uCUoyPe7xi+ZQksHPSmlYx9pDm8Ndbkx3OwbxSrKUTG88ru227Er78aqQZI7H0uRIwauKZZ6BG3lAg0b4=@freebsd.org X-Gm-Message-State: AOJu0YxA+Uk5luRhrfJ76igzZCHh/WMI5eiGDY1rlZBBmf2xMYpWgmSb wPtSUqcAUCQYVybEF4sxGtgZrPwREq2NIAZ6vObaw0zqz6jCypoyjqLNlCky8HHbsbArUhIHJe1 Za/HiAmYrhgwwgI+w1tdNcyKlv4ljvk8= X-Gm-Gg: ASbGncuGRiGzCGw89F8H2ETdHRxnFxYSpwzgVmaVQ36Vuf2YbILdBD2+eq+8NbsFLnu CG94OfUQieIQ3azhIAjxyv89zE+Hs1GHR3DwUWvOdrPfjcLgOuxWI8xkr4LOV8bOPKeYJr+BBhj EUrjkQum1rJEgxUMRpw9M2rKdj4/MLx9LWivPWw6XidZdkSIAm4fpKV60WpmCNYdTnn6sEgFklo IBGV0nMHcoLkRrqpynE9r6KiKP5DiEh2za837skE/wt1gRBJIQgP9lIKP7kqG3C18rqKbxdQmei I7qHjIOENGs9Zs68dg1yJsqVEA== X-Google-Smtp-Source: AGHT+IGY20EDj3BGrMXtgqhfKGWEActLHuGo0nzjCIE4v0gUsOWcP7xh8cLMEtGetMGd96XPXVNzA6TNvl1z1F4lz60= X-Received: by 2002:a05:6402:50ce:b0:641:66cc:9d91 with SMTP id 4fb4d7f45d1cf-6491a90f87emr3707450a12.27.1765103453545; Sun, 07 Dec 2025 02:30:53 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 References: <18FB2858-5CBB-4B7A-8089-224A58C6A160@yahoo.com> In-Reply-To: From: Mateusz Guzik Date: Sun, 7 Dec 2025 11:30:41 +0100 X-Gm-Features: AQt7F2quTV5J9LTiRJ1Ah9ZssWWckX4XMkKMeXv99BU9HgE-68Qn8aNnkbJw_Vg Message-ID: Subject: Re: performance regressions in 15.0 To: Warner Losh Cc: Mark Millard , FreeBSD Current , FreeBSD-STABLE Mailing List Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4dPLxJ06M7z46sD On Sat, Dec 6, 2025 at 11:26=E2=80=AFPM Warner Losh wrote: > A few months before I landed the jemalloc patches, i did 4 or 5 from dirt= buildworlds. The elasped time was, iirc, with 1 or 2%. Enough to see maybe= a diff with the small sample size, but not enough for ministat to trigger = at 95%. I didn't recall keeping the data for this and can't find it now. An= d I'm not even sure, in hindsight, I ran a good experiment. It might be rel= ated, or not, but it would be easy enough for someone to setup a two jails:= one just before and one just after. Build from scratch the world (same has= h) on both. That would test it since you'd be holding all other variables c= onstant. > > When we imported the tip of FreeBSD main at work, we didn't get a cpu cha= nge trigger from our tests that I recall... > Note you probably build tested with clang which was already penalized. I just verified that going to libc as of this commit: commit c43cad87172039ccf38172129c79755ea79e6102 (HEAD) Merge: da260ab23f26 48ec896efb0b Author: Warner Losh Date: Mon Aug 11 17:38:36 2025 -0600 jemalloc: Merge from jemalloc 5.3.0 vendor branch retains the perf problem as seen in the malloc microbenchmarks and that going to one commit prior bring it back in line with 14.3 built like so from lib/libc: make -s -j 8 WITHOUT_TESTS=3D1 MALLOC_PRODUCTION=3Dyes all install Given that jemalloc prior to the import is a well known working state, I think it will be most prudent to revert the update for the time being and investigate it later. Note both jemalloc itself and clang aside, there is the issue of slower binary startup in the first place (see the doexec.c parts in my e-mail). Given the magnitude of the slowdowns, the above two are definitely EN material. Sorting out the startup thing should qualify depending on complexity of the fix, whatever it might be.