[Bug 288149] Need a way to balance MAKE_JOBS_NUMBER to available RAM and enforce max parallelity

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=288149

            Bug ID: 288149
           Summary: Need a way to balance MAKE_JOBS_NUMBER to available
                    RAM and enforce max parallelity
           Product: Ports & Packages
           Version: Latest
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: Ports Framework
          Assignee: portmgr@FreeBSD.org
          Reporter: mandree@FreeBSD.org
                CC: ports-bugs@FreeBSD.org

We have numerous issues in building ports that arise out of an imbalance of
parallel builders versus the RAM use per builder, or because ports disregard
job limits imposed through MAKE_JOBS_NUMBER.

Beyond that, we see LTO options spread that are extremely memory hungry.

We end up with various issues where parallel and/or LTO builds fail with
SIGKILL because we exhaust available RAM on the builder or address space is
exhausted on 32-bit computers.

Random examples:
* d4f780099d800b87701e4341a68666b318219097 - lang/swift510 built with number of
cores instead of a lower MAKE_JOBS_NUMBER.
* https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=288131 - cargo-c cannot
build on a 8 GB RAM computer. some architectures disable LTO already. 12 GB RAM
and 5 jobs are fine.
* we've had to disable LTO from Python builds on a low rental root server
because LTO builds wouldn't fit in 3 GB RAM on a single-core computer
* https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277529 - go doesn't respect
MAKE_JOBS_NUMBER=1
* https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=287976 - emacs runs into
SIGSEGV on memory exhaustion, usual cause is unguarded memory allocation that
doesn't handle errors such a NULL returns from allocation functions.

We need to develop a way to flag or enforce builders that ramp parallelity to
high, and we should start monitoring max RAM use as a function of parallelity
(MAKE_JOB_NUMBERS) and LTO flag (and possibly other memory hogs) and then take
action.

Possible approaches are giving a number of "GB basic need plus GB per socket"
or some function which can be set by ports so that the framework can assess the
number of cores and GB of RAM and limit parallelity accordingly.

-- 
You are receiving this mail because:
You are on the CC list for the bug.