poudriere 3.1.15 and 3.1.16 builders get stuck in {balancing,clean}_pool status

[Naram Qashat]

I have been transitioning to using poudriere to build ports for my 
system with custom options. Everything has been working well until I 
updated poudriere to 3.1.15 and then later 3.1.16.

My host is running FreeBSD 11.0-RELEASE-p8 amd64, and the poudriere jail 
is running the same version (using the /usr/src / /usr/obj from the 
host). The host has a 6-core Intel i7 (so FreeBSD treats this at 12 CPU 
cores).

When I start a bulk build, I notice that sometimes, after a port has 
finished building, the status of a builder will go to balancing_pool or 
clean_pool instead of idle. Some of the ports that were built will then 
show up as failed AND built, with the error in the failed section 
showing a phase of balancing_pool and the build log showing "Build 
crashed: balancing_pool" near the very end. Eventually it gets to a 
state where poudriere no longer continues building ports, with all the 
builders in either the balancing_pool status or clean_pool status.

I noticed this originally with poudriere 3.1.15 when attempting to just 
bulk build poudriere alone (to get 3.1.16), but at that time I didn't 
notice what state the builder was in before I killed the bulk build. I 
had to update poudriere by building from the port's directory directly. 
Then when I was on poudriere 3.1.16 and started a brand new bulk build, 
that is when I noticed the balancing_pool and clean_pool statuses.

This doesn't happen if I downgrade back to poudriere 3.1.14 (using 
portdowngrade to grab a copy of the port from before 3.1.15).

The relevant changes in my poudriere.conf from the sample version:

* Not using ZFS (NO_ZFS=yes)
* Using portlint (USE_PORTLINT=yes)
* CHECK_CHANGED_OPTIONS set to verbose
* CHECK_CHANGED_DEPS set to yes
* Setting a PKG_REPO_SIGNING_KEY to use the packages built by poudriere 
through pkg
* Only using 4 builders instead of the default 12 (PARALLEL_JOBS=4)
* Save failed WRKDIRs (SAVE_WRKDIR=yes)
* Only allowing MAKE_JOBS on chromium, clang* and llvm* 
(ALLOW_MAKE_JOBS_PACKAGES="chromium clang* llvm*")
* Timestamp my logs (TIMESTAMP_LOGS=yes)

I believe I created the poudriere setup with 3.1.14 or maybe even 
3.1.13. (It was only created within the last couple weeks.)

If more information is needed to diagnose this, please let me know.

Thanks,
Naram Qashat