For comparison, head -r338860 based -j4 buildworld buildkernel time and swap usage on Pine64+ 2GB

Sun Sep 23 23:20:04 UTC 2018

As I remember Bob Prohaska's fastest -j4 buildworld buildkernel
full-builds on his rpi3 in his recent experiments were
somewhat over 22 hours and had fairly extensive swapping.

This report allows comparison to a particular Pine64+ 2GB
based -j4 buildworld buildkernel, although I give details Bob
P. did not. I also was not running any monitoring other than
a modified top, not logged to a file.

/usr/obj/cortexA53_clang_alt did not exist when this started,
so this is for a from-scratch build.

(The below avoids some >>> prefix line text.)

QUOTE
Script started on Sun Sep 23 00:28:14 2018
Command: env __MAKE_CONF=/root/src.configs/make.conf SRCCONF=/dev/null SRC_ENV_CONF=/root/src.configs/src.conf.cortexA53-clang-bootstrap.aarch64-host WITH_META_MODE=yes MAKEOBJDIRPREFIX=/usr/obj/cortexA53_clang_alt/arm64.aarch64 make -j4 buildworld buildkernel
[Creating objdir /usr/obj/cortexA53_clang_alt/arm64.aarch64/usr/src/arm64.aarch64...]
--- buildworld ---
make[1]: "/usr/src/Makefile.inc1" line 341: SYSTEM_COMPILER: Determined that CC=cc matches the source tree.  Not bootstrapping a cross-compiler.
make[1]: "/usr/src/Makefile.inc1" line 346: SYSTEM_LINKER: Determined that LD=ld matches the source tree.  Not bootstrapping a cross-linker.
--- buildworld_prologue ---
--------------------------------------------------------------
 World build started on Sun Sep 23 00:28:16 PDT 2018
--------------------------------------------------------------
. . .
--------------------------------------------------------------
 Kernel build for GENERIC-NODBG completed on Sun Sep 23 14:11:05 PDT 2018
--------------------------------------------------------------

Command exit status: 0
Script done on Sun Sep 23 14:11:05 2018
END QUOTE

So somewhat under 13 hours 45 minutes when the bootstrap avoided the
cross-compiler and cross-linker builds when I'm building -r338860
via selecting:

#WITH_CROSS_COMPILER=
WITH_SYSTEM_COMPILER=
WITH_SYSTEM_LINKER=
#
#CPUTYPE=soft
WITH_LIBCPLUSPLUS=
#WITH_LLD_BOOTSTRAP=
WITHOUT_BINUTILS_BOOTSTRAP=
WITH_ELFTOOLCHAIN_BOOTSTRAP=
#WITH_CLANG_BOOTSTRAP=
WITH_CLANG=
WITH_CLANG_IS_CC=
WITH_CLANG_FULL=
WITH_CLANG_EXTRAS=
WITH_LLD=
WITH_LLD_IS_LD=
WITHOUT_BINUTILS=
WITH_LLDB=
#
WITH_BOOT=
WITHOUT_LIB32=
#
WITHOUT_GCC_BOOTSTRAP=
WITHOUT_GCC=
WITHOUT_GCC_IS_CC=
WITHOUT_GNUCXX=

MALLOC_PRODUCTION=
#
WITH_REPRODUCIBLE_BUILD=
WITH_DEBUG_FILES=

When I build for cortex-A53's I use:

CFLAGS.clang+= -mcpu=cortex-a53
CXXFLAGS.clang+= -mcpu=cortex-a53
CPPFLAGS.clang+= -mcpu=cortex-a53
ACFLAGS.arm64cpuid.S+=  -mcpu=cortex-a53+crypto
ACFLAGS.aesv8-armx.S+=  -mcpu=cortex-a53+crypto
ACFLAGS.ghashv8-armx.S+=        -mcpu=cortex-a53+crypto

The environment that did the build had been built that
way as well. Similarly for the GENERIC-NODBG used:

# more /usr/src/sys/arm64/conf/GENERIC-NODBG
#
# GENERIC -- Custom configuration for the arm64/aarch64
#

include "GENERIC"

ident   GENERIC-NODBG

makeoptions     DEBUG=-g                # Build kernel with gdb(1) debug symbols

options         ALT_BREAK_TO_DEBUGGER

options         KDB                     # Enable kernel debugger support

# For minimum debugger support (stable branch) use:
#options        KDB_TRACE               # Print a stack trace for a panic
options         DDB                     # Enable the kernel debugger

# Extra stuff:
#options        VERBOSE_SYSINIT         # Enable verbose sysinit messages
#options        BOOTVERBOSE=1
#options        BOOTHOWTO=RB_VERBOSE
#options        KTR
#options        KTR_MASK=KTR_TRAP
##options       KTR_CPUMASK=0xF
#options        KTR_VERBOSE

# Disable any extra checking for. . .
nooptions       DEADLKRES               # Enable the deadlock resolver
nooptions       INVARIANTS              # Enable calls of extra sanity checking
nooptions       INVARIANT_SUPPORT       # Extra sanity checks of internal structures, required by INVARIANTS
nooptions       WITNESS                 # Enable checks to detect deadlocks and cycles
nooptions       WITNESS_SKIPSPIN        # Don't run witness on spinlocks for speed
nooptions       DIAGNOSTIC
nooptions       MALLOC_DEBUG_MAXZONES   # Separate malloc(9) zones
nooptions       BUF_TRACKING
nooptions       FULL_BUF_TRACKING

I've modified top in my active environments to report:

Maximum Observed Mem Active (reported: 1501M)
Maximum Observed Swap Used  (reported:  198M)
Minimum Observed Mem Free   (reported:   33M)

After the buildworld buildkernel completed top was showing:

Mem: 79M Active, 1501M MaxObsActive, 1352M Inact, 1056K Laundry, 347M Wired, 201M Buf, 194M Free, 33M MinObsFree
Swap: 3584M Total, 19M Used, 198M MaxObsUsed, 3565M Free

I had enabled using and used an e.MMC in DDR52 mode on a microsd
adapter, using TRIM on the UFS file system (with the DISCARD bit
set). I also had to enable DISCARD. "Enable(d)" here means
source code changes in both cases.

No other types of storage devices were connected so swap was on
the same device as the UFS file system partition. After the
build:

# df -m
Filesystem               1M-blocks  Used Avail Capacity  Mounted on
/dev/label/PINE64P2Groot    109101 50910 49462    51%    /
devfs                            0     0     0   100%    /dev
/dev/label/PINE642GAboot        63     8    55    13%    /boot/efi

I have in /boot/loader.conf :

vfs.ffs.dotrimcons=1

and in /etc/sysctl.conf :

vm.pageout_oom_seq=120
vfs.ffs.dotrimcons=1

(But vfs.ffs.dotrimcons defaults to 1 now as I remember.
As I understand vm.pageout_oom_seq still defaults to
12. 120 might not be sufficient for a rpi3 to buildworld
buildkernel in some contexts, or even for the Pine64+
2GB if the storage I/O is sufficiently problematical for
latencies that occur.)

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)