GSoC: Separation of Ports Build Process from Local Installation

[Theron Tarigo]

Hello all,

This project aims to decouple the build process of the ports framework 
from the local installation.  To work around the assumptions made by 
ports about dependency file locations, a userspace tool to remap 
processes' filesystem namespaces is under development: 
freebsd-user-namespace, provided as port devel/userns.

This past week, I have made the following progress:
- Eliminated the need for any special setup to be able to use 
PORTS_SEPARATED_BUILD.  The mode may be enabled by this one variable, at 
the command line or in /etc/make.conf.
- Created a lightweight framework for testing large numbers of ports 
under the separated build mode.  It does the following:
       - Handles FLAVORS.
       - Saves success/failure/skipped status.
       - Skips ports which have failed dependencies.
       - Builds each port in its own pristine environment.
         (This allows concurrent building of unrelated ports)
       - Saves success/failure/skipped status.
       - Skips ports which have failed dependencies.
       - Builds each port in its own pristine environment.
         (This allows parallel (concurrent) port build to work)
- Attempted build of 2000 ports, selected at random, and their dependencies.
- Identified (and applied temporary fixes for) some ports causing 
highest number of skipped dependents.
- Retried the skipped ports.
- Developed a plan of a general fix for the most common cause of 
failure: custom target definitions in port Makefiles running build tools 
(also from ports) without the userns wrapper.

Latest summary of port building success:
- 5613 ports tried
- 2958 (53%) succeeded
- 177 (3%) failed
- 2478 (44%) skipped due to failed dependencies
- Success rate: 94%

What I am working on next:

- Implement the planned solution for running build tools inside userns 
wrapper.
- Compare the packages built by separated-build method to either 
official packages or ports built by original FreeBSD method.  Any 
differences should be attributable to built times and hostnames.
- Debug the ports which fail by segfaut or buserror; this implicates 
userns library.
- Provide more thorough documentation of PORTS_SEPARATED_BUILD, in 
particular its usage and its limitations, so that users and devs 
interested in this project can more easily test it for themselves.

Project goals and status are kept at 
https://wiki.freebsd.org/SummerOfCode2019Projects/PortsSeparatedBuild .

Source of the userspace filesystem namespace tool is shared at 
https://github.com/therontarigo/freebsd-user-namespace .
Changes to ports framework are shared at 
https://github.com/freebsd/freebsd-ports/compare/master...therontarigo:separated.

Theron Tarigo