poudriere 3.1.15 and 3.1.16 builders get stuck in {balancing,clean}_pool status

[Bryan Drewery]

On 3/22/2017 9:45 AM, Naram Qashat wrote:
> I have been transitioning to using poudriere to build ports for my
> system with custom options. Everything has been working well until I
> updated poudriere to 3.1.15 and then later 3.1.16.

Thanks for your great report and sorry for the trouble!

> 
> My host is running FreeBSD 11.0-RELEASE-p8 amd64, and the poudriere jail
> is running the same version (using the /usr/src / /usr/obj from the
> host). The host has a 6-core Intel i7 (so FreeBSD treats this at 12 CPU
> cores).
> 
> When I start a bulk build, I notice that sometimes, after a port has
> finished building, the status of a builder will go to balancing_pool or
> clean_pool instead of idle. Some of the ports that were built will then
> show up as failed AND built, with the error in the failed section
> showing a phase of balancing_pool and the build log showing "Build
> crashed: balancing_pool" near the very end. Eventually it gets to a
> state where poudriere no longer continues building ports, with all the
> builders in either the balancing_pool status or clean_pool status.
> 
> I noticed this originally with poudriere 3.1.15 when attempting to just
> bulk build poudriere alone (to get 3.1.16), but at that time I didn't
> notice what state the builder was in before I killed the bulk build. I
> had to update poudriere by building from the port's directory directly.
> Then when I was on poudriere 3.1.16 and started a brand new bulk build,
> that is when I noticed the balancing_pool and clean_pool statuses.
> 
> This doesn't happen if I downgrade back to poudriere 3.1.14 (using
> portdowngrade to grab a copy of the port from before 3.1.15).
> 
> The relevant changes in my poudriere.conf from the sample version:
> 
> * Not using ZFS (NO_ZFS=yes)
> * Using portlint (USE_PORTLINT=yes)
> * CHECK_CHANGED_OPTIONS set to verbose
> * CHECK_CHANGED_DEPS set to yes
> * Setting a PKG_REPO_SIGNING_KEY to use the packages built by poudriere
> through pkg
> * Only using 4 builders instead of the default 12 (PARALLEL_JOBS=4)
> * Save failed WRKDIRs (SAVE_WRKDIR=yes)
> * Only allowing MAKE_JOBS on chromium, clang* and llvm*
> (ALLOW_MAKE_JOBS_PACKAGES="chromium clang* llvm*")
> * Timestamp my logs (TIMESTAMP_LOGS=yes)

This is the problem here, TIMESTAMP_LOGS=yes. Set it to no for now.

A fix is ready for release in 3.1.17.  I am just waiting on some other
things and more time before releasing 3.1.17.

If you depend on this functionality the patch to fix it is:

https://people.freebsd.org/~bdrewery/patches/timestamp-exit-on-eof.diff

> 
> I believe I created the poudriere setup with 3.1.14 or maybe even
> 3.1.13. (It was only created within the last couple weeks.)
> 
> If more information is needed to diagnose this, please let me know.
> 
> Thanks,
> Naram Qashat


-- 
Regards,
Bryan Drewery

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-ports/attachments/20170322/8fd4dd4e/attachment.sig>