[ Week Report GSOC ] Parallelization in the ports collection

Alexander Pronin scher at FreeBSD.org
Tue Jun 12 14:07:10 UTC 2012


LOCK_DIR changed from ${PORTSDIR}/lock to /var/db/portlocks.

Make process (submake) is allowed to work in locked port's if and only if this port's dir was locked by parent make process. Hence all submakes can work in port's dir, locked by parent make.

First parallel non-blocking build of port's dependecies as background jobs in XXX-depends targets. Tested on fake ports.
_parv_PARALLEL_BUILDS_NUMBER variable controls number of parallel builds. May be set by user to choose degree of parallelization.

 ${_PROCESS_ACTIVE_BUILDS}: new script that covers most of non-blocking parallel dependency build magic.
        Controls all spawned background jobs by PIDs which are stored in $${active_builds} array
        Controls jobs exit codes.
        Acts accordingly on various jobs exit codes (0, 1, ${_parv_MAKE_LOCK_EXIT_STATUS}).
        Recheck correct install of lib dependency.
        Considers parallel dep builds limit as set by ${_parv_PARALLEL_BUILDS_NUMBER}.
        Waits for a while if it is impossible to spawn new dependency build.
        Makes attempts to spawn another dependency build, if it is possible.

Changes in old targets (XXX-depends):
        Reimplementation of loops in XXX-depends targets for non-blocking purpose:
        Outer "while" loop replaced "for" loop.
        Some kind of "round-robin" implementation is used to process port's deps.
        If some dependency (dep) is locked then it is moved to the end of the deps queue.
        If some dep is already installed then it is deleted from the deps queue.
        If some dependency is not found - it's build starts as background process and
        it's PID is added to $${active_builds} array to control it's evaluation.

All the above mentioned is actual for non-parallel builds, with the following exception:                 
${_parv_PARALLEL_BUILDS_NUMBER} is set to 1 in non-parallel case. This perevents from spawning more than one background job. Thus processing of port's deps will be blocked until this job will be ready. Hence we have one background job per time for non-parallel build.

Correct termination of the whole process tree with background processes starting from parent make process. To terminate the whole process tree, starting from some parent make process (pm_pid) one may just use keyboard interrupts in the right terminal (where the make process was started) or send signal to internal make script using pkill(1). e.g. pkill -P pm_pid



More information about the soc-status mailing list