Re: devel/llvm13 failed to reclaim memory on 8 GB Pi4 running -current

On 2022-Jun-4, at 14:49, Marcin Cieslak <saper@saper.info> wrote:

> On Fri, 28 Jan 2022, Mark Millard wrote:
> 
> [ Reviving old thread ]
> 
>> After that I intend runs with 30 GiBytes of swap (so RAM+SWAP 38 GiBytes).
>> Hopefully that will complete and I'll be able to report how much swap was
>> observed to have been used.
> 
> I thought I will get LLVM14 + OpenJDK built quickly so I fired up
> a c6g.4xlarge AWS instance (16 vCPUs, 32 GB RAM),
> or even c6g.8xlarge (32 vCPUs, 64 GB RAM) and even these
> are unable to build llvm14 under FreeBSD 13.1-RELEASE
> with poudrière enabling MAKE_JOBS for llvm build.

It depends on the build options. Do you need fortran,
a.k.a. flang ?

Building flang requires building MLIR because building
flang uses MLIR. Without needing flang, both can normally
be turned off.

flang does not build for armv7 and other 32-bit environments,
so recently the /usr/ports/devel/llvm14/Makefile was modified
to not have flang as a default for armv7 (or armv6). My
understanding, the intent is to later also turn off MLIR
for these.

I have built devel/llvm14 with FLANG and MLIR disabled on
a 8 GiByte RPi4B in an armv7 poudriere jail. This was
as part of an ongoing "bulk -a -c" on the RPi4B. (I will
not be able to let it run to completion but am testing
how things go until I stop it.)  The below notes are
somewhat biased by my also having used BE_NATIVE for the
devel/llvm14 build:

---Begin OPTIONS List---
===> The following configuration options are available for llvm14-14.0.2:
    BE_AMDGPU=on: AMD GPU backend (required by mesa)
    BE_WASM=on: WebAssembly backend (required by firefox via wasi)
    CLANG=on: Build clang
    DOCS=on: Build and/or install documentation
    EXTRAS=on: Extra clang tools
    LIT=on: Install lit and FileCheck test tools
    LLD=on: Install lld, the LLVM linker
    LLDB=on: Install lldb, the LLVM debugger
    MLIR=off: Multi-Level Intermediate Representation
    PYCLANG=on: Install python bindings to libclang
====> Options available for the single BACKENDS: you have to select exactly one of them
    BE_FREEBSD=off: Backends for FreeBSD architectures
    BE_NATIVE=on: Backend(s) for this architecture (ARM)
    BE_STANDARD=off: All non-experimental backends
===> Use 'make config' to modify these settings
---End OPTIONS List---

The RPi4B has 4 cores and I had poudriere using 4 builders
and ALLOW_MAKE_JOBS= with no explicit constraint on the make
jobs per builder, so implicitly 4. Thus it can have periods
with load averages around 16 when when things do as I said
to do. (Some ports do not limit themselves to the HWthread
count.) Result:

[54:41:45] [01] [13:40:48] Finished devel/llvm14 | llvm14-14.0.2: Success

Note that that is with 2 GiBytes of RAM per core, the
same ratio that your report indicates.

I will also report that I used a UFS context, not ZFS,
and that my patched top indicated little swap usage
during any of my "bulk -a -c" atttempt so far. (The
top tracks and reports various MaxObs???? figures,
i.e., "Maximum Observed" figures.)

My environments do have /boot/loader.conf ,
/etc/sysctl.conf , and /usr/local/etc/poudriere.conf
settings appropriate to avoiding OOM kills for long
running [but bounded duration] build activity for the
hardware that I have access to --and that is also
part of why I prefer UFS for 2 GiBytes/HWthread for
doing builds.

(I do have access to 3 systems with 4 GiBytes
per HWthread and on 2 of them I normally use
ZFS, including for build activity.)

> The build proceeds on 1 CPU only now

Such was not true of my build reported above.
Using one core at a time, the RPi4B would have
taken much longer to do the build.

> - and casual observation
> with top confirms that compilation of certain C++ files
> requires 7..8 GB of RAM.

My understanding is that such has a known status for
flang/MLIR being in use.

> If 16 of them are built concurrently,
> no wonder that there is not enough memory.
> 
> Files tha crashed my compiliation:
> 
> /wrkdirs/usr/ports/devel/llvm14/work/llvm-project-14.0.4.src/flang/lib/Evaluate/tools.cpp
> /wrkdirs/usr/ports/devel/llvm14/work/llvm-project-14.0.4.src/clang/lib/Sema/SemaOpenMP.cpp
> /wrkdirs/usr/ports/devel/llvm14/work/llvm-project-14.0.4.src/flang/lib/Semantics/check-omp-structure.cpp
> /wrkdirs/usr/ports/devel/llvm14/work/llvm-project-14.0.4.src/flang/lib/Evaluate/fold-integer.cpp

You do not report the related console messages related (or
dmsg -a output or /var/log/messages content). I can think
of multiple distinct possibilities for the reports that
would have different implications. (But such need not be
of any importance for you.)

I see 3 /flang/ paths in the list of 4 paths, not surprising.

(Your /usr/ports/ context is more recent than mine.)

> Sure, poudrière could get Kubernetes-like capabilities to control
> resources during the build one day, but aren't amounts of memory
> needed to build the compiler excessive a bit?

"the compiler" ignores a fair mount of what is built.
Part of what I'm suggesting above is to configure
the devel/llvm14 build to build fewer compilers/toolchains
(no flang, no MLIR). You might not want, say, EXTRAS, or
some other things I have "on". BE_NATIVE may not fit
your usage context well but BE_FREEBSD could be considered.
And so on.

Hopefully my notes above are of some use unless
you are required to use the default OPTIONS.


Side note:

The RPi4B bulk actually built llvm13 and rust with
a vast part of the time frames overlapping and other
things going through the other 2 builders.

It later worked on llvm12, ghc, and suitesparse-graphblas
with the vast part of the time being overlapped. Again:
all 4 builders doing something. (suitesparse-graphblas
does not last as many hours as the other 2.)

No kernel OOM process kills. Under 300 MiByte swap used
at any point.

(ghc's haddock runs out of 32-bit process memory in
a normal core-dump way, towards the end the ghc build,
so ghc failed. But the build had been going for most
of 24 hours before that point, as had llvm12.)

===
Mark Millard
marklmi at yahoo.com