Re: git: 636592343c3e - main - tmpfs: increase memory reserve to a percent of available memory + swap

Baptiste Daroussin <bapt_at_freebsd.org> wrote on
Date: Mon, 22 Jan 2024 13:23:37 UTC :

> On Mon, Jan 22, 2024 at 07:15:22AM -0600, Mike Karels wrote:
> > On 21 Jan 2024, at 21:48, Alexey Dokuchaev wrote:
> > 
> > > On Mon, Jan 22, 2024 at 03:27:57AM +0000, Jessica Clarke wrote:
> > >> On 19 Dec 2023, at 15:34, Mike Karels <karels@FreeBSD.org> wrote:
> > >>> commit 636592343c3ec0feb61a4d8043676381384420dd
> > >>>
> > >>> tmpfs: increase memory reserve to a percent of available memory + swap
> > >>>
> > >>> [...]
> > >>>
> > >>> Add sysctl for vfs.tmpfs.memory_percent and the pre-existing
> > >>> vfs.tmpfs.memory_reserved to tmpfs(5).
> > >>>
> > >>> PR: 275436
> > >>> MFC after: 1 month
> > >>> Reviewed by: rgrimes
> > >>> Differential Revision: https://reviews.freebsd.org/D43011
> > >>
> > >> I just got:
> > >>
> > >> make[6]: Could not create temporary file /tmp/makeDl4i9S:
> > >> No space left on device
> > >>
> > >> after doing a buildworld -j4 on a 4 core, 16 GiB system where /tmp is a
> > >> tmpfs, with the prior buildworld having taken me past this commit,
> > >> which I strongly suspect to be the cause. Whilst I understand the
> > >> motivation behind your commits, this is a sad regression to have; it's
> > >> not exactly a particularly strange situation.
> > 
> > Bug 276402 is about the interaction of this change with ZFS ARC, which
> > seems to be problematical. I spent some time investigating yesterday.
> > Depending on the dynamics, I get different results. If tmpfs grows,
> > the ARC will be reduced if needed. But if the ARC grows to the point
> > that tmpfs sets free space to 0 or quite low, attempts to write to
> > tmpfs fail without any reduction in ARC.
> > 
> > Jess, two quesions:
> > 
> > 1. Are you using ZFS on this system?
> > 
> > 2. Can you try with vfs.tmpfs.memory_percent set to 100?
> > 
> > > FWIW, I've seen two people complain about this last week, apparently
> > > this kernel OOM protection doesn't work very well. :-/
> > 
> > So far, I have only seen OOM kills when memory + swap are quite short,
> > with a tmpfs memory_percent above 95. But with ZFS, it can happen
> > when the ARC could have been reduced.
> > 
> > I will investigate more today. If I don't find a way to resolve this,
> > I'll probably commit a change to set vfs.tmpfs.memory_percent to 100
> > by default, at least for now, and if that works around the problem.
> > 
> This is easily reproducible with poudriere, if you try to build some
> ports/packages which are big memory consumers, for example any chrome variant or
> just lang/rust, I have a machine with 64G of ram

How many hardware threads?
How much swap space?
You specified: USE_TMPFS=all
ALLOW_MAKE_JOBS=yes ?
MAKE_JOBS_NUMBER use (or the like)?
I assume ZFS but any ARC specific tuning?

Not that you would want to do what I do, but it
illustrates why I ask about the contexts explored.


16 hardware thread aarch64 with 64 GiBytes of RAM context:

I've historically used:
ZFS untuned ARC context (but I also do the same in UFS contexts)
USE_TMPFS=all
ALLOW_MAKE_JOBS=yes
no MAKE_JOBS_NUMBER other than for editors/lapce*
RAM+SWAP == 310 GiBytes (so a little over RAM+SWAP==4.8*RAM)

Basically SWAP is preventing tmpfs use from running
out of space, despite the likes of 25 GiByte+ use
of tmpfs by the rust build, for example.

Historically multiple toolchains building in parallel
in the resulting high load average context have
completed just fine, including when rust is one of
them. I've not come close to running out of RAM+SWAP
so far.

Although my testing of "bulk -a" is very rare, such
has worked when I tried it, also using the hig load
average style.

Why ~4.8? Because somewhere between there and 5.0 the
binding of the swap space starts puttting out a notice
about potentially being mistuned. (It reports on a
potential change that I do not know the other tradeoffs
for. So I stay in the range where no notice is made.)

> and it is impossible to get
> lang/rust build in poudriere (USE_TMPFS=all) without setting
> vfs.tmpfs.memory_percent to 100.

I'm guessing this is implicitly: without having a huge
RAM+SWAP space.

> the poudriere dies with plenty of no space left on device.

I will note that I've not tested my configuration
with the change yet, holding back on updating FreeBSD
for other reasons. But I doubt that I'd run out of
RAM+SWAP, given the huge RAM+SWAP that I've
historically used for the context.

I do analogously on 2 other systems:
32  GiBytes of RAM and  8 hardware threads
192 GiBytes of RAM and 32 hardware threads
(Both having both ZFS and UFS media.)

On the various small RAM systems, I use
USE_TMPFS=data or USE_TMPFS=no and avoid
ZFS.

I also always avoid spinning rust.


===
Mark Millard
marklmi at yahoo.com