Re: Resolved: devel/llvm13 build: "ninja: build stopped: subcommand failed"

On 2022-Aug-15, at 06:05, Nuno Teixeira <eduardo@freebsd.org> wrote:

> Hi Mark,
> 
> I use TMPFS=no

That is the wrong notation. Quoting /usr/local/etc/poudriere.conf.sample :

QUOTE
# Use tmpfs(5)
# This can be a space-separated list of options:
# wrkdir    - Use tmpfs(5) for port building WRKDIRPREFIX
# data      - Use tmpfs(5) for poudriere cache/temp build data
# localbase - Use tmpfs(5) for LOCALBASE (installing ports for packaging/testing)
# all       - Run the entire build in memory, including builder jails.
# yes       - Enables tmpfs(5) for wrkdir and data
# no        - Disable use of tmpfs(5)
# EXAMPLE: USE_TMPFS="wrkdir data"
USE_TMPFS=yes
END QUOTE

Note: involving wrkdir uses a lot of tmpfs for some
ports.

So a correct notation in /usr/local/etc/poudriere.conf
would be to replace the USE_TMPFS=yes assignment with:

USE_TMPFS=no

This might be why something is competing for RAM+SWAP
such that you run out: You might have wrkdir involved
via an implicit use of yes if you actualyl used the
notation TMPFS=no instead.

While RAM+SWAP use is high (but not yet out of space),
you can use a command like:

# df -m | egrep "^(Filesystem|tmpfs) "
Filesystem                                                             1M-blocks Used   Avail Capacity  Mounted on
tmpfs                                                                       1024   35     988     3%    /usr/local/poudriere/data/.m/main-CA72-bulk_a-default/ref/.p
tmpfs                                                                      31790    0   31790     0%    /usr/local/poudriere/data/.m/main-CA72-bulk_a-default/ref/var/db/ports
tmpfs                                                                       1024    0    1023     0%    /usr/local/poudriere/data/.m/main-CA72-bulk_a-default/01/.p

to get a clue how much tmpfs is in use to see if it
is large relative to your RAM+SWAP space.

> and I don't have WITH_DEBUG set.

Good.

> I will test with MAKE_JOBS_NUMBER=n in /usr/local/etc/poudriere.d/make.conf and see what max n I can get it to compile and observe used ram+swap.
> 
> Other possibility is to increase swap to <=60GB but I'd like to avoid that because I will need to resize freebsd-zfs partition.
> What you think about increasing swap?

As I mentioned before, you can add SWAP without modifying
your existing media by adding new media that has another
freebsd-swap partition and can set up to have the old and
new freebsd-swap partitions both be active at the same
time, thereby increasing the total SWAP. So, for example,
a USB3 NVMe SSD could be used to increase the SWAP space
available. (You may well have other reasons to not want
to do such a thing. But it is technically possible to do.)

But, hopefully, you can find and fix why something extra is
competing for your RAM+SWAP and so avoid needing more SWAP
space.

> Thanks
> 
> Mark Millard <marklmi@yahoo.com> escreveu no dia segunda, 15/08/2022 à(s) 04:26:
> On 2022-Aug-14, at 20:06, Tomoaki AOKI <junchoon@dec.sakura.ne.jp> wrote:
> 
> > On Sun, 14 Aug 2022 12:23:03 -0700
> > Mark Millard <marklmi@yahoo.com> wrote:
> > 
> >> On 2022-Aug-14, at 11:07, Nuno Teixeira <eduardo@freebsd.org> wrote:
> >> 
> >>> I will follow https://docs.freebsd.org/en/books/handbook/book/#disks-growing and resize actual swap, but before that I will have to make sure that backups are ok in case of something goes wrong.
> >>> 
> >>> I've tooked a note about total swap <=60GB
> >>> 
> >>> . . .
> >> 
> >> 
> >> I forgot an important question relative to the resource use
> >> for building devel/llvm13 and later: do you need/want the
> >> fortran compiler?
> >> 
> >> If not, you can disable the options: FLANG MLIR
> >> (building FLAG would require building MLIR)
> >> 
> >> Then the build will be far less memory intensive
> >> and take less time.
> >> 
> >> It had slipped my mind that my builds have these 2 options
> >> disabled.
> >> 
> >> ===
> >> Mark Millard
> >> marklmi at yahoo.com
> > 
> > Is there any possibility that something alike Bug 264949 [1] for gcc11
> > is happening?
> > 
> > For amd64, option GOLD (LTO support) is implied by default.
> > If it causes LTO to be enabled for llvm13 build itself, and lld act as
> > linker of gcc11, small TMPDIR would overflow.
> > 
> > gcc11 required at minimum 5GiB of free space on TMPDIR.
> > (4GiB was insufficient.)
> > 
> > [1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264949
> > 
> 
> Nuno had a specific problem others have not reported:
> 
> QUOTE
> I've tested it but it still fails:
> ---
> pid 64502 (c++), jid 7, uid 65534, was killed: failed to reclaim memory
> swap_pager: out of swap space
> ---
> on a Lenovo Legion 5, 16GB RAM and 4GB swap.
> END QUOTE
> 
> "was killed: failed to reclaim memory" and "swap_pager: out of swap
> space" are not about "small TMPDIR overflow", at least not directly.
> 
> But if the tmpfs is of a form backed by RAM+SWAP, then the tmpfs
> use can grow to contribute to causing reclaim problems and/or out of
> swap space problems by competing for RAM+SWAP space.
> 
> Nuno reported having disabled such tmpfs use ( via USE_TMPFS=no in
> /usr/local/etc/poudriere.conf ). I do not know if that status is
> well confirmed or not. I've also no clue if WITH_DEBUG= might be
> in use. (WITH_DEBUG needs to not be in use unless huge resources are
> available.)
> 
> My amd64 tests indicate that something beyond the compiles/links
> themselves was using significant RAM+SWAP in Nuno's context. I've
> no clue what.



===
Mark Millard
marklmi at yahoo.com