RPI3 swap experiments [a failure for a Pine64+ 2GB at head -r337400 without vm.pageout_oom_seq or #define adjustment, no I/O latency problem reported]

Mon Aug 13 10:17:00 UTC 2018

[vm.pageout_oom_seq=120 worked in this Pine64+ 2GB environment.]

On 2018-Aug-12, at 10:59 AM, Mark Millard <marklmi at yahoo.com> wrote:

> [Quick top post: Possibly related bugzilla's for this area
> are:
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227609
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230402
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230454
> .]
> 
> On 2018-Aug-12, at 10:15 AM, Mark Millard <marklmi at yahoo.com> wrote:
> 
>> Based on having all Mark Johnston's reporting-patches but
>> not adjusting vm.pageout_oom_seq or the #define I got
>> an OOM kill during buildworld on a Pine64+ 2GB (so 2 GiByte
>> of RAM).
>> 
>> I'll give other environment characteristic later.
>> But that no I/O latency problems were reported by the
>> patches is not a surprise: the device with the root file
>> system and swap partition has not historically shown
>> latency problems and is not of a type usually used with
>> such a system (better than normal).
>> 
>> I expect that this case shows that the problem does not
>> require I/O latency problems to be involved.
>> 
>> The only console message was:
>> 
>> Aug 12 09:30:13 pine64 kernel: v_free_count: 7773, v_inactive_count: 1
>> Aug 12 09:30:13 pine64 kernel: pid 80573 (c++), uid 0, was killed: out of swap space
>> 
>> The build had been started at: 01:44:59 so the failure
>> was around 7 hours 45 minutes into the build.
>> 
>> The build log shows:
>> 
>> Building /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/clang/libclang/Sema/SemaDeclAttr.o
>> Building /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/clang/libclang/Sema/SemaDeclCXX.o
>> Building /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/clang/libclang/Sema/SemaDeclObjC.o
>> --- Sema/SemaDecl.o ---
>> c++: error: unable to execute command: Killed
>> c++: error: clang frontend command failed due to signal (use -v to see invocation)
>> FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
>> Target: aarch64-unknown-freebsd12.0
>> Thread model: posix
>> InstalledDir: /usr/bin
>> c++: note: diagnostic msg: PLEASE submit a bug report to https://bugs.freebsd.org/submit/ and include the crash backtrace, preprocessed source, and associated run script.
>> c++: note: diagnostic msg: 
>> ********************
>> 
>> PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
>> Preprocessed source(s) and associated run script(s) are located at:
>> c++: note: diagnostic msg: /tmp/SemaDecl-44b104.cpp
>> c++: note: diagnostic msg: /tmp/SemaDecl-44b104.sh
>> c++: note: diagnostic msg: 
>> 
>> ********************
>> *** [Sema/SemaDecl.o] Error code 254
>> 
>> 
>> Description of the context . . .
>> 
>> I have access to a Pine64+ 2GB (that was updated to -r337400
>> via a cross build) and had it doing a self-hosted rebuild of
>> -r337400 via -j4. (This was a jump from its last update over
>> 10 months ago. I've not historically had OOM kill problems.)
>> 
>> The root file system is on a USB3.0-capable 240 GB SSD
>> plugged in the lower slot, where it gets the 480Mbps
>> USB 2.0 classification. The kernel was loaded from a
>> microSDHC card that also supplied a /etc/fstab that
>> redirects to the USB root file system.
>> 
>> (With an edit of that /etc/fstab the microSDHC can boot
>> standalone: I keep it tracking.)
>> 
>> # usbconfig
>> ugen0.1: <Allwinner EHCI root HUB> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (0mA)
>> ugen1.1: <Generic OHCI root HUB> at usbus1, cfg=0 md=HOST spd=FULL (12Mbps) pwr=SAVE (0mA)
>> ugen0.2: <OWC Envoy Pro mini> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (0mA)
>> 
>> Historically I've not observed latency problems for
>> the USB SSD. None were reported by Mark Johnston's
>> reporting patches, unlike for Bob P.'s context.
>> 
>> In top I'd seem over 1400 Mem Active, well over the amount
>> of RAM in a rpi3 or rpi2. (Of course with more memory the
>> relative timings of what is running will be different
>> from what rpi3's/rpi2's get. That is not the only source of
>> variation.)
>> 
>> Swap had been used, I've seen 19M Used shown (of 3 GiByte
>> in the swap partition in use). (I do not have in place my
>> prior changes to record and report the maximum observed
>> swap used: I reverted to the normal version of top during
>> top's development activity.)
>> 
>> I'm unlikely to have observed approximate the maximums for
>> Mem Active or Swap Used. These are things I wish there were
>> normally available for inspection from FreeBSD.
>> 
>> I did not have a parallel loop showing gstat output
>> or other such, relying on Mark Johnston's patches
>> for reporting any that are a problem for the subsystem(s)
>> involved. (It is also less I/O to not be running the
>> extra logs.)
>> 

vm.pageout_oom_seq=120 worked in this Pine64+ 2GB environment.
There were no messages from Mark Johnston's reporting patches.

It took a little under 13 hr 40 minutes for a from-scratch
build that did not build the bootstrap compiler or bootstrap
linker but did build the clang extras:

# ~/sys_build_scripts.aarch64-host/make_cortexA53_nodebug_clang_bootstrap-aarch64-host.sh -j4 buildworld buildkernel
. . .
--- buildworld ---
make[1]: "/usr/src/Makefile.inc1" line 341: SYSTEM_COMPILER: Determined that CC=cc matches the source tree.  Not bootstrapping a cross-compiler.
make[1]: "/usr/src/Makefile.inc1" line 346: SYSTEM_LINKER: Determined that LD=ld matches the source tree.  Not bootstrapping a cross-linker.
. . .
--------------------------------------------------------------
>>> Kernel build for GENERIC-NODBG completed on Mon Aug 13 00:44:10 PDT 2018
--------------------------------------------------------------

Script done, output file is /root/sys_typescripts/typescript_make_cortexA53_nodebug_clang_bootstrap-aarch64-host-2018-08-12:11:07:29

(The script file name has the start time in it.)

That was based on:

# more ~/src.configs/src.conf.cortexA53-clang-bootstrap.aarch64-host 
TO_TYPE=aarch64
#
KERNCONF=GENERIC-NODBG
TARGET=arm64
.if ${.MAKE.LEVEL} == 0
TARGET_ARCH=${TO_TYPE}
.export TARGET_ARCH
.endif
#
#WITH_CROSS_COMPILER=
WITH_SYSTEM_COMPILER=
WITH_SYSTEM_LINKER=
#
#CPUTYPE=soft
WITH_LIBCPLUSPLUS=
#WITH_LLD_BOOTSTRAP=
WITHOUT_BINUTILS_BOOTSTRAP=
WITH_ELFTOOLCHAIN_BOOTSTRAP=
#WITH_CLANG_BOOTSTRAP=
WITH_CLANG=
WITH_CLANG_IS_CC=
WITH_CLANG_FULL=
WITH_CLANG_EXTRAS=
WITH_LLD=
WITH_LLD_IS_LD=
WITHOUT_BINUTILS=
WITH_LLDB=
#
WITH_BOOT=
WITHOUT_LIB32=
#
WITHOUT_GCC_BOOTSTRAP=
WITHOUT_GCC=
WITHOUT_GCC_IS_CC=
WITHOUT_GNUCXX=
#
NO_WERROR=
#WERROR=
MALLOC_PRODUCTION=
#
WITH_REPRODUCIBLE_BUILD=
WITH_DEBUG_FILES=
#
# Use of the .clang 's here avoids
# interfering with other C<?>FLAGS
# usage, such as ?= usage.
CFLAGS.clang+= -mcpu=cortex-a53
CXXFLAGS.clang+= -mcpu=cortex-a53
CPPFLAGS.clang+= -mcpu=cortex-a53
ACFLAGS.arm64cpuid.S+=  -mcpu=cortex-a53+crypto
ACFLAGS.aesv8-armx.S+=  -mcpu=cortex-a53+crypto
ACFLAGS.ghashv8-armx.S+=        -mcpu=cortex-a53+crypto

I used the following:

# more monitor_ram_swap.sh 
#!/bin/sh
while true
do  sleep 1 ; date ; top -CawSores | egrep "(Mem|Swap):"
done 

to monitor part of the build, although I accidentally
stopped it part way through clang/libclang being built
and did not notice at the time. This was somewhat past
where the prior vm.pageout_oom_seq=12 test had the
supposedly-OOM kill.

The peak observed mem-active was 1572M:
(I manually added the blank lines before date output.)

. . .
Sun Aug 12 19:12:27 PDT 2018
Mem: 1473M Active, 64M Inact, 752K Laundry, 352M Wired, 202M Buf, 76M Free
Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse

Sun Aug 12 19:12:28 PDT 2018
Mem: 1478M Active, 65M Inact, 752K Laundry, 352M Wired, 202M Buf, 69M Free
Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse

Sun Aug 12 19:12:29 PDT 2018
Mem: 1540M Active, 39M Inact, 752K Laundry, 351M Wired, 202M Buf, 35M Free
Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse

Sun Aug 12 19:12:31 PDT 2018
Mem: 1568M Active, 12M Inact, 1192K Laundry, 342M Wired, 202M Buf, 41M Free
Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse

Sun Aug 12 19:12:32 PDT 2018
Mem: 1572M Active, 2780K Inact, 1512K Laundry, 342M Wired, 202M Buf, 49M Free
Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse

Sun Aug 12 19:12:33 PDT 2018
Mem: 1113M Active, 3828K Inact, 1612K Laundry, 342M Wired, 202M Buf, 505M Free
Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse
. . .

But this was not where the (earlier) peak observed swap-use of 222M Used
was recorded:

. . .
Sun Aug 12 18:56:35 PDT 2018
Mem: 1118M Active, 209M Inact, 250M Laundry, 345M Wired, 202M Buf, 43M Free
Swap: 3072M Total, 216M Used, 2856M Free, 7% Inuse

Sun Aug 12 18:56:36 PDT 2018
Mem: 1117M Active, 209M Inact, 250M Laundry, 345M Wired, 202M Buf, 44M Free
Swap: 3072M Total, 216M Used, 2856M Free, 7% Inuse

Sun Aug 12 18:56:37 PDT 2018
Mem: 1123M Active, 208M Inact, 247M Laundry, 345M Wired, 202M Buf, 42M Free
Swap: 3072M Total, 219M Used, 2853M Free, 7% Inuse

. . . (an omitted block of 219M Used examples) . . .

Sun Aug 12 18:57:09 PDT 2018
Mem: 1114M Active, 217M Inact, 247M Laundry, 346M Wired, 202M Buf, 41M Free
Swap: 3072M Total, 219M Used, 2853M Free, 7% Inuse

Sun Aug 12 18:57:10 PDT 2018
Mem: 1127M Active, 205M Inact, 244M Laundry, 347M Wired, 203M Buf, 42M Free
Swap: 3072M Total, 222M Used, 2850M Free, 7% Inuse

Sun Aug 12 18:57:11 PDT 2018
Mem: 988M Active, 162M Inact, 148M Laundry, 345M Wired, 202M Buf, 320M Free
Swap: 3072M Total, 56M Used, 3016M Free, 1% Inuse
. . .

The first and last recorded was:

Sun Aug 12 11:24:50 PDT 2018
Mem: 62M Active, 435M Inact, 4596K Laundry, 454M Wired, 202M Buf, 1014M Free
Swap: 3072M Total, 19M Used, 3053M Free
. . .
Sun Aug 12 19:47:06 PDT 2018
Mem: 782M Active, 83M Inact, 1068K Laundry, 347M Wired, 202M Buf, 753M Free
Swap: 3072M Total, 41M Used, 3031M Free, 1% Inuse

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)