Re: Restarting poudriere

On Tue, 15 Jun 2021 01:31:32 -0700
Mark Millard <marklmi@yahoo.com> wrote:

> Michael Gmelin <freebsd_at_grem.de> wrote on
> Date: Tue, 15 Jun 2021 09:31:26 +0200 :
> 
> On Mon, 14 Jun 2021 18:02:33 -0700
> bob prohaska <fbsd_at_www.zefox.net> wrote:
> >   
> > > On Mon, Jun 14, 2021 at 09:52:22PM +0200, Michael Gmelin wrote:  
> > > > 
> > > >     
> > > > > On 14. Jun 2021, at 20:30, bob prohaska
> > > > > <fbsd_at_www.zefox.net> wrote:
> > > > > 
> > > > > ???On Mon, Jun 14, 2021 at 06:46:52PM +0200, Michael Gmelin
> > > > > wrote:    
> > > > >> On Mon, 14 Jun 2021 09:28:39 -0700
> > > > >> What do you mean by "restarting"?
> > > > >> How do you invoke poudriere exactly?    
> > > > > As root,
> > > > > poudriere bulk -J 2 -j main x11-wm/lxqt www/chromium >
> > > > > testbuild.log    
> > > > 
> > > > Doesn???t ninja handle parallel builds on its own anyway? Does
> > > > it work okay if you comment out ALLOW_*_JOBS in poudriere.conf?
> > > >    
> > > 
> > > The line ALLOW_MAKE_JOBS is already commented out in 
> > > /usr/local/etc/poudriere.conf  but it's active in
> > > /etc/make.conf
> > > 
> > > I remain a bit confused about how poudriere and make coordinate
> > > their parallel job spawning activity. In the latest case the -J 2
> > > on the poudriere command line put two packages under
> > > construction, but the ALLOW_MAKE_JOBS line in /etc/make.conf
> > > didn't result in parallel threads building LLVM10. Clearly I
> > > don't understand the relationship between builders, jobs, threads
> > > and (much!) more.   
> > 
> > /etc/make.conf from the host system is not used by poudriere, as it
> > aims to provide clean-room builds and therefore doesn't use
> > configurations that also serve a purpose on the host system.
> > 
> > All configurations come from /usr/local/etc/poudriere.conf and
> > various files in /usr/local/etc/poudriere.d/.  
> 
> I'll explicitly note that one of the files is:
> 
> /usr/local/etc/poudriere.d/make.conf
> 
> It is the most global make.conf in poudriere's activities.
> For poudriere, someone writing about using make.conf likely
> is implicitly referencing the one at that path (unless they
> report something that indicates otherwise).
> 
> Bob: Sorry I did not notice the /etc/make.conf reference in
> your earlier material. I should have commented on the
> distinction.
> 
> > See also the
> > CUSTOMISATION section in poudriere(8) (reading that man pages is
> > useful in general).
> > 
> > So if you want to use this feature, you need to enable it in
> > poudriere.conf (but I would still recommend using
> > ALLOW_MAKE_JOBS_PACKAGES instead and also only if you don't use
> > multiple builders, given the limited resources of your machine).  
> 
> /usr/local/etc/poudriere.d/make.conf use should work fine
> as well.

Nope, ALLOW_MAKE_JOBS_PACKAGES is a poudriere setting, used
here inside poudriere:
https://github.com/freebsd/poudriere/blob/master/src/share/poudriere/common.sh#L4421

poudriere doesn't read make.conf configurations as its own
configuration, so setting ALLOW_MAKE_JOBS_PACKAGES in there does
nothing (it only appends make.conf settings to the
generated /etc/make.conf inside the jail, see "append_make()" in
common.sh).

Also note that ALLOW_MAKE_JOBS_PACKAGES has no effect for ports
tested using "poudriere testport" (but it *does* apply to building
the dependencies required for such ports). So, assuming
ALLOW_MAKE_JOBS_PACKAGE="someport", "poudriere testport someport" won't
make use of MAKE_JOBS, while "poudriere testport
otherport-that-depends-on-someport" will do so for building someport.

Something that might not be obvious when reading poudriere's config:
/usr/local/etc/poudriere.d/poudriere.conf is only read if
/usr/local/etc/poudriere.conf doesn't exist. Other files like
/usr/local/etc/poudriere.d/130amd64-poudriere.conf *will* be read
though (after reading the global config).

Cheers,
Michael

> 
> My guess is that Bob will explore what various combinations
> do, including for hitting resource limits vs. not. He has a
> long history of doing such and I've learned some (for me)
> handy things from some of his explorations and what they
> lead to.
> 
> > > Perhaps I've complicated matters by using testport on earlier,
> > > successful, tries. Is there a significant difference if one just
> > > wants to make  packages? I merely wanted to see how the
> > > attempt would fail and hoped for more diagnostics. But it worked.
> > > So, I tried bulk and that doesn't seem to work so well.  
> > 
> > As someone pointed out earlier in this thread, looking at the
> > poudriere build log of a failed build (and maybe sharing it) could
> > help to find a solution.  
> 
> 
> Bob has a history of posting large log files someplace for
> reference. So I'd not be surprised if he does that again for
> this issue at some point.
> 
> ===
> Mark Millard
> marklmi at yahoo.com
> ( dsl-only.net went
> away in early 2018-Mar)



-- 
Michael Gmelin