After update to r357104 build of poudriere jail fails with 'out of swap space'

[Cy Schubert]

On January 27, 2020 10:19:50 AM PST, "Rodney W. Grimes" <freebsd-rwg at gndrsh.dnsmgr.net> wrote:
>> In message <202001261745.00QHjkuW044006 at gndrsh.dnsmgr.net>, "Rodney
>W. 
>> Grimes"
>> writes:
>> > > In message <20200125233116.GA49916 at troutmask.apl.washington.edu>,
>Steve 
>> > > Kargl w
>> > > rites:
>> > > > On Sat, Jan 25, 2020 at 02:09:29PM -0800, Cy Schubert wrote:
>> > > > > On January 25, 2020 1:52:03 PM PST, Steve Kargl
><sgk at troutmask.apl.wash
>> > ingt
>> > > > on.edu> wrote:
>> > > > > >On Sat, Jan 25, 2020 at 01:41:16PM -0800, Cy Schubert wrote:
>> > > > > >> 
>> > > > > >> It's not just poudeiere. Standard port builds of chromium,
>rust
>> > > > > >> and thunderbird also fail on my machines with less than 8
>GB.
>> > > > > >>
>> > > > > >
>> > > > > >Interesting.  I routinely build chromium, rust, firefox,
>> > > > > >llvm and few other resource-hunger ports on a i386-freebsd
>> > > > > >laptop with 3.4 GB available memory.  This is done with
>> > > > > >chrome running with a few tabs swallowing a 1-1.5 GB of
>> > > > > >memory.  No issues.  
>> > > > > 
>> > > > > Number of threads makes a difference too. How many
>core/threads does yo
>> > ur l
>> > > > aptop have?
>> > > >
>> > > > 2 cores.
>> > > 
>> > > This is why.
>> > > 
>> > > >
>> > > > > Reducing number of concurrent threads allowed my builds to
>complete
>> > > > > on the 5 GB machine. My build machines have 4 cores, 1 thread
>per
>> > > > > core. Reducing concurrent threads circumvented the issue. 
>> > > >
>> > > > I use portmaster, and AFIACT, it uses 'make -j 2' for the
>build.
>> > > > Laptop isn't doing too much, but an update and browsing.  It
>does
>> > > > take a long time especially if building llvm is required.
>> > > 
>> > > I use portmaster as well (for quick incidental builds). It uses 
>> > > MAKE_JOBS_NUMBER=4 (which is equivalent to make -j 4). I suppose
>machines 
>> > > with not enough memory to support their cores with certain builds
>might 
>> > > have a better chance of having this problem.
>> > > 
>> > > MAKE_JOBS_NUMBER_LIMIT to limit a 4 core machine with less than 2
>GB per 
>> > > core might be an option. Looking at it this way, instead of an
>extra 3 GB, 
>> > > the extra 60% more memory in the other machine makes a big
>difference. A 
>> > > rule of thumb would probably be, have ~ 2 GB RAM for every core
>or thread 
>> > > when doing large parallel builds.
>> >
>> > Perhaps we need to redo some boot time calculations, for one the
>> > ZFS arch cache, IMHO, is just silly at a fixed percent of total
>> > memory.  A high percentage at that.
>> >
>> > One idea based on what you just said might be:
>> >
>> > percore_memory_reserve = 2G (Your number, I personally would use 1G
>here)
>> > arc_max = MAX(memory size - (Cores * percore_memory_reserve),
>512mb)
>> >
>> > I think that simple change would go a long ways to cutting down the
>> > number of OOM reports we see.  ALSO IMHO there should be a way for
>> > sub systems to easily tell zfs they are memory pigs too and need to
>> > share the space.  Ie, bhyve is horrible if you do not tune zfs arc
>> > based on how much memory your using up for VM's.
>> >
>> > Another formulation might be
>> > percore_memory_reserve = alpha * memory_zire / cores
>> >
>> > Alpha most likely falling in the 0.25 to 0.5 range, I think this
>one
>> > would have better scalability, would need to run some numbers.
>> > Probably needs to become non linear above some core count.
>> 
>> Setting a lower arc_max at boot is unlikely to help. Rust was
>building on 
>> the 8 GB and 5 GB 4 core machines last night. It completed
>successfully on 
>> the 8 GB machine, while using 12 MB of swap. ARC was at 1307 MB.
>> 
>> On the 5 GB 4 core machine the rust build died of OOM. 328 KB swap
>was 
>> used. ARC was reported at 941 MB. arc_min on this machine is 489.2
>MB.
>
>What is arc_max?   
>
>> Cy Schubert <Cy.Schubert at cschubert.com>

3.8 GB. It never exceeds 1.5 to 2 GB when doing a NO_CLEAN buildworld, where it gets a 95-99% hit ratio with 8 threads.

There are a couple of things going on here. First, four large multithreaded rust compiles in memory simultaneously. Secondly, a reluctance to use swap. My guess is the working set for each of the four compiles was large enough to trigger the OOM.  I haven't had time to seriously look at this though but I'm guessing that the locality of reference was large enough to keep much of the memory in RAM, so here we are.




-- 
Pardon the typos and autocorrect, small keyboard in use. 
Cy Schubert <Cy.Schubert at cschubert.com>
FreeBSD UNIX: <cy at FreeBSD.org> Web: https://www.FreeBSD.org

The need of the many outweighs the greed of the few.

Sent from my Android device with K-9 Mail. Please excuse my brevity.