Re: Controlling python when building www/chromium

From: bob prohaska <fbsd_at_www.zefox.net>
Date: Tue, 24 Aug 2021 08:35:54 -0700
On Tue, Aug 24, 2021 at 11:09:54AM +0200, Ronald Klop wrote:
> 
> Van: Mark Millard via freebsd-ports <freebsd-ports_at_freebsd.org>
> Datum: dinsdag, 24 augustus 2021 10:58
> Aan: freebsd-ports_at_freebsd.org, bob prohaska <fbsd_at_www.zefox.net>
> Onderwerp: Re: Controlling python when building www/chromium
> > 
> > 
> > bob prohaska <fbsd_at_www.zefox.net> wrote on
> > Date: Mon, 23 Aug 2021 08:27:33 -0700 :
> > 
> > > When trying to build www/chromium on a 1 GB Pi3 the system
> > > gets bogged down by five  instances of python2.7 running
> > > simultaneously. This happens using both poudriere and make.
> > >
> > > It wasn't a problem a year ago, so presumably something has
> > > changed in chromium's internal build machinery. I've searched
> > >
> > > https://www.chromium.org/developers/how-tos/get-the-code
> > >  and
> > >
> > > https://groups.google.com/a/chromium.org/g/chromium-dev
> > >
> > > but couldn't find any references to python when compiling, only
> > > when running the browser.  >
> > > Is there some way to control how many pythons are loosed at one time?
> > > Most likely two could be accomodated, possibly three. On an 8 GB
> > > Pi4 the five pythons coexist happily, so the behavior is probably
> > > not considered a bug.
> > 
> > 
> > Bob did not show the context. Below I show an example from his
> > public poudriere logs, a copy from an off-list mail, for
> > reference:
> > 
> > QUOTE
> > When I looked recently, the peak swap usage reported was:
> > 
> > Fri Aug  6 00:39:58 PDT 2021
> > Device          1K-blocks     Used    Avail Capacity
> > /dev/da0s2b       1843200  1617020   226180    88%
> > /dev/mmcsd0s2b    1843200  1615244   227956    88%
> > Total             3686400  3232264   454136    88%
> > 
> > and was for (showing the one after that total):
> > 
> > `-- /bin/sh ./buildscript.chromium
> >  `-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium
> >    |-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium
> >    |-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium
> >    `-- sh: poudriere[main-default][01]: build_pkg (chromium-91.0.4472.114_1) (sh)
> >      |-- sh: poudriere[main-default][01]: build_pkg (chromium-91.0.4472.114_1) (sh)
> >      | `-- /usr/bin/make -C /usr/ports/www/chromium build
> >      |   `-- (sh)
> >      |     `-- ninja -j1 -C out/Release chromedriver -v chrome
> >      |       `-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . .
> >      |         |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . .
> >      |         |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . .
> >      |         |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . .
> >      |         `-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . .
> >      `-- timestamp
> > END QUOTE
> > 
> > This was the most extreme swap/paging space usage from somewhat
> > analogous use of a generate_bindings. The swap/paging space
> > usage makes trying multiple builders impractical: it actually
> > does run out of swap/paging space. (There are limits to how
> > big of a swap avoids potential mistuning for a given size RAM.
> > swap/paging+RAM can be larger on a 8 GiByte RPi4B can be much
> > larger than on a RPi3B, without getting notices suggesting
> > a mistuned environment.)
> > 
> > (It is not necessarily Python 2.7. The build, overall, only uses
> > 2.7 sometimes in some places.)
> > 
> > https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/blink/renderer/bindings/scripts/generate_bindings.py shows:
> > 
> > 
> >     bind_gen.init(web_idl_database_path=options.web_idl_database,
> >                   root_src_dir=options.root_src_dir,
> >                   root_gen_dir=options.root_gen_dir,
> >                   component_reldirs=component_reldirs,
> >                   enable_style_format=options.format_generated_files)
> >     task_queue = bind_gen.TaskQueue(single_process=options.single_process)
> >     for task in options.tasks:
> >         dispatch_table[task](task_queue)
> > 
> > which I would guess is the code initiating the parallel python
> > processes above.
> > 
> > Looking at the history, the first use of .TaskQueue here seems to have
> > been at:
> > 
> > chromium / chromium / src / 3b8f5a3b2903f2aa50efb971f430ddce57a17d16^! / . / third_party / blink / renderer / bindings / scripts / generate_bindings.py
> > commit  3b8f5a3b2903f2aa50efb971f430ddce57a17d16    [log] [tgz]
> > author  Yuki Shiino <yukishiino_at_chromium.org>   Thu Jun 04 05:01:43 2020
> > committer   Commit Bot <commit-bot_at_chromium.org>    Thu Jun 04 05:01:43 2020
> > tree    46e7ea9138d3452f1b864cbe1d9c3d1adaa62a57
> > parent  54c74ea75c7984bbfa11ed6232821af1943ef922 [diff] [blame]
> > 
> > 
> > (I did not look for other contexts with .TaskQueue uage additions.)
> > 
> > 
> > 
> > ===
> > Mark Millard
> > marklmi at yahoo.com
> > ( dsl-only.net went
> > away in early 2018-Mar)
> > 
> > 
> > 
> > 
> 
> 
> Hi,
> 
> Without looking into the code, I would guess that it should be possible to set "options.single_process" (if -j1 is set).
> As the RPI3 has 4 cores it apparently auto-scales to the amount of CPUs.
> But I don't have the time to look into the ninja build file.
> 

After the latest poudriere failure I again tried using make in
www/chromium. It also got bogged down with five pythons and
eventually stopped. At that point I simply restarted the make
session without cleaning and left the system to run overnight.

As of this morning there are two c++ instances, the system is
only ~45% idle and it seems to be making good progress:[ 18% 5544/29872]

The two c++ instances are consistent with /etc/make.conf .

Does this behavior offer any hints into when and how the surplus
of pythons gets hatched? The build session is visible via
http://www.zefox.org/~bob/usr.ports/www/chromium/

The actvivity logging script is still running from the poudriere
session and the output is visible at
http://www.zefox.org/~bob/swaplogs/20210816.log
however the file is now somewhat unwieldy in size.

Thanks for reading, and any thoughts. 

bob prohaska
Received on Tue Aug 24 2021 - 15:35:54 UTC

Original text of this message