Re: Very long-running buildworld process

From: Mark Millard <marklmi_at_yahoo.com>
Date: Wed, 11 Jun 2025 05:24:26 UTC
On Jun 10, 2025, at 20:59, bob prohaska <fbsd@www.zefox.net> wrote:

> Buildworld for -current on a Pi2B is strangely slow. The machine is responsive,
> but seems to make no (or very slow) progress:
> 
> 
> login: Jun  7 22:51:31 www su[3032]: bob to root on /dev/pts/0
> 
> 
> FreeBSD/arm (www.zefox.org) (ttyu0)
> 
> login: bob
> Password:
> Last login: Sat Jun  7 22:51:25 from gateway.zefox.net
> FreeBSD 15.0-CURRENT #85 main-d8773fdcbfa3: Fri Jun  6 14:17:58 PDT 2025     bob@www.zefox.org:/usr/obj/usr/src/arm.armv7/sys/GENERIC
> 
> Welcome to FreeBSD!
> 
> Release Notes, Errata: https://www.FreeBSD.org/releases/
> Security Advisories:   https://www.FreeBSD.org/security/
> last pid: 34625;  load averages:    1.01,    1.01,    1.00                                                             up 2+22:03:17  20:45:47
> last pid: 34628;  load averages:    1.07,    1.02,    1.01                                                             up 2+22:05:21  20:47:51
> 45 processes:  2 running, 43 sleeping
> CPU: 25.0% user,  0.0% nice,  0.0% system,  0.0% interrupt, 75.0% idle
> Mem: 6196K Active, 297M Inact, 174M Wired, 98M Buf, 443M Free
> Swap: 1770M Total, 422M Used, 1348M Free, 23% Inuse

Free  443M
Inact 297M
Wired 174M

Yet:

c++ RES is just 17M and Active is just 6196K.

Also: 422M SWAP used.

I suggest sorting the top output by (decreasing) RES and
seeing if that shows what is contributing to Inact 297M
as part of RESident memory use.

Any chance of tmpfs use or other significant competition
for RAM+SWAP?

Also, figuring out what is using the SWAP space could be
important. (It might not the same.)

Also, top can display more of the command lines. With such
one can more about what the c++ command involved is.

>  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
> 21654 root          1 135    0   561M    17M CPU1     1  39.8H 100.12% c++
> 34626 bob           1  20    0  6664K  3064K CPU3     3   0:00   0.31% top
> 2843 root          1  20    0  4684K   756K select   2   2:07   0.01% powerd
> 10915 root          1  20    0  7036K   824K select   1   0:08   0.00% make
> 2934 root          1  28    0    13M  2408K select   0   4:39   0.00% sshd
> 16200 root          1  20    0    11M  3908K select   1   0:39   0.00% make
> 2951 root          1  20    0    11M  1836K select   0   0:25   0.00% sendmail
> 2719 root          1  20    0  5608K  1292K kqread   2   0:12   0.00% syslogd
> 3085 root          1  20    0  6340K   864K select   0   0:08   0.00% make
> 11703 root          1  20    0  5968K   816K select   3   0:08   0.00% make
> 3044 root          1  20    0  5920K   848K select   1   0:08   0.00% make
> 10901 root          1  20    0  6340K   824K select   0   0:08   0.00% make
> 9496 root          1  20    0  6340K   820K select   0   0:08   0.00% make
> 2946 root          1  20    0  5216K   944K nanslp   1   0:07   0.00% cron
> 
> Note the 38 hour runtime for PID 21654...
> 
> Buildworld output ends with:
> ...
> Building /usr/obj/usr/src/arm.armv7/lib/clang/libclang/Tooling/JSONCompilationDatabase.pico
> Building /usr/obj/usr/src/arm.armv7/lib/clang/libclang/Tooling/Refactoring.pico
> Building /usr/obj/usr/src/arm.armv7/lib/clang/libclang/Tooling/RefactoringCallbacks.pico
> Building /usr/obj/usr/src/arm.armv7/lib/clang/libclang/Tooling/Tooling.pico
> root@www:/usr/src #

Note all those ".pico" references. The build environment has had
a significant change. Its consequences for small RAM armv7 or
aarch64 boards need not be obvious:

QUOTE
The branch main has been updated by dim:

URL: https://cgit.FreeBSD.org/src/commit/?id=2e47f35be5dc61945afdbd1a70e8fd505c032c94

commit 2e47f35be5dc61945afdbd1a70e8fd505c032c94
Author: Dimitry Andric <dim@FreeBSD.org>
AuthorDate: 2025-04-25 17:56:39 +0000
Commit: Dimitry Andric <dim@FreeBSD.org>
CommitDate: 2025-04-25 17:56:39 +0000

Convert libllvm, libclang and liblldb into private shared libraries

This allows clang, lld, lldb, and other llvm tools to be linked against
these shared libraries, which makes them smaller and avoids duplication.

Since these are not quite the same as the shared libraries shipped by
the upstream llvm build system, and we do not want to expose the ABI to
external programs such as ports, make them private libraries.

Note that during the cross-tools stage they are still built as static
libraries, so the cross compiler and linker are static binaries, as they
have always been.

This also requires a depend-cleanup.sh kludge which will be added in a
follow-up commit, to ensure binaries are rebuilt against the shared
libraries in case of incremental builds.
END QUOTE

> Likely the culprit is one of those jobs.

linking may be more resource intensive now, for all I know.

> Gstat reports occasional brief access to /dev/da0s2a, both the serial console
> and the controlling ssh session for buildworld are responsive, the host isn't
> crashed.
> 
> The buildworld command is
> make -j4 -DWITH_META_MODE  KERNCONF=GENERIC  TARGET=arm TARGET_ARCH=armv7 buildworld > buildworld.log

Your:

last pid: 34628;  load averages:    1.07,    1.02,    1.01 
with:
CPU: 25.0% user,  0.0% nice,  0.0% system,  0.0% interrupt, 75.0% idle

suggests generally 3 idle FreeBSD cpus at a time and one active.
So for some reason -j4 did not have work to do for more than one
make job at that time --and the job was not multithreading to
have more cpus active.

Avoiding multithreaded linking on 32 bit systems is normal because
of bigger problems fitting in the 32 bit address space otherwise.

> Is there a graceful way to figure out what's going on?





===
Mark Millard
marklmi at yahoo.com