Re: Controlling python when building www/chromium

From: Ronald Klop <ronald-lists_at_klop.ws>
Date: Tue, 24 Aug 2021 11:09:54 +0200 (CEST)
 
Van: Mark Millard via freebsd-ports <freebsd-ports_at_freebsd.org>
Datum: dinsdag, 24 augustus 2021 10:58
Aan: freebsd-ports_at_freebsd.org, bob prohaska <fbsd_at_www.zefox.net>
Onderwerp: Re: Controlling python when building www/chromium
> 
> 
> bob prohaska <fbsd_at_www.zefox.net> wrote on
> Date: Mon, 23 Aug 2021 08:27:33 -0700 :
> 
> > When trying to build www/chromium on a 1 GB Pi3 the system
> > gets bogged down by five  instances of python2.7 running
> > simultaneously. This happens using both poudriere and make.
> >
> > It wasn't a problem a year ago, so presumably something has
> > changed in chromium's internal build machinery. I've searched
> >
> > https://www.chromium.org/developers/how-tos/get-the-code
> >  and
> >
> > https://groups.google.com/a/chromium.org/g/chromium-dev
> >
> > but couldn't find any references to python when compiling, only
> > when running the browser.  
> >
> > Is there some way to control how many pythons are loosed at one time?
> > Most likely two could be accomodated, possibly three. On an 8 GB
> > Pi4 the five pythons coexist happily, so the behavior is probably
> > not considered a bug.  
> 
> 
> Bob did not show the context. Below I show an example from his
> public poudriere logs, a copy from an off-list mail, for
> reference:
> 
> QUOTE
> When I looked recently, the peak swap usage reported was:
> 
> Fri Aug  6 00:39:58 PDT 2021
> Device          1K-blocks     Used    Avail Capacity
> /dev/da0s2b       1843200  1617020   226180    88%
> /dev/mmcsd0s2b    1843200  1615244   227956    88%
> Total             3686400  3232264   454136    88%
> 
> and was for (showing the one after that total):
> 
> `-- /bin/sh ./buildscript.chromium
>  `-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium
>    |-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium
>    |-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium
>    `-- sh: poudriere[main-default][01]: build_pkg (chromium-91.0.4472.114_1) (sh)
>      |-- sh: poudriere[main-default][01]: build_pkg (chromium-91.0.4472.114_1) (sh)
>      | `-- /usr/bin/make -C /usr/ports/www/chromium build
>      |   `-- (sh)
>      |     `-- ninja -j1 -C out/Release chromedriver -v chrome
>      |       `-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . .
>      |         |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . .
>      |         |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . .
>      |         |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . .
>      |         `-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . .
>      `-- timestamp
> END QUOTE
> 
> This was the most extreme swap/paging space usage from somewhat
> analogous use of a generate_bindings. The swap/paging space
> usage makes trying multiple builders impractical: it actually
> does run out of swap/paging space. (There are limits to how
> big of a swap avoids potential mistuning for a given size RAM.
> swap/paging+RAM can be larger on a 8 GiByte RPi4B can be much
> larger than on a RPi3B, without getting notices suggesting
> a mistuned environment.)
> 
> (It is not necessarily Python 2.7. The build, overall, only uses
> 2.7 sometimes in some places.)
> 
> https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/blink/renderer/bindings/scripts/generate_bindings.py shows:
> 
> 
>     bind_gen.init(web_idl_database_path=options.web_idl_database,
>                   root_src_dir=options.root_src_dir,
>                   root_gen_dir=options.root_gen_dir,
>                   component_reldirs=component_reldirs,
>                   enable_style_format=options.format_generated_files)
>     task_queue = bind_gen.TaskQueue(single_process=options.single_process)
>     for task in options.tasks:
>         dispatch_table[task](task_queue)
> 
> which I would guess is the code initiating the parallel python
> processes above.
> 
> Looking at the history, the first use of .TaskQueue here seems to have
> been at:
> 
> chromium / chromium / src / 3b8f5a3b2903f2aa50efb971f430ddce57a17d16^! / . / third_party / blink / renderer / bindings / scripts / generate_bindings.py
> commit  3b8f5a3b2903f2aa50efb971f430ddce57a17d16    [log] [tgz]
> author  Yuki Shiino <yukishiino_at_chromium.org>   Thu Jun 04 05:01:43 2020
> committer   Commit Bot <commit-bot_at_chromium.org>    Thu Jun 04 05:01:43 2020
> tree    46e7ea9138d3452f1b864cbe1d9c3d1adaa62a57
> parent  54c74ea75c7984bbfa11ed6232821af1943ef922 [diff] [blame]
> 
> 
> (I did not look for other contexts with .TaskQueue uage additions.)
> 
> 
> 
> ===
> Mark Millard
> marklmi at yahoo.com
> ( dsl-only.net went
> away in early 2018-Mar)
> 
>  
> 
> 
> 


Hi,

Without looking into the code, I would guess that it should be possible to set "options.single_process" (if -j1 is set).
As the RPI3 has 4 cores it apparently auto-scales to the amount of CPUs.
But I don't have the time to look into the ninja build file.

Regards,
Ronald.
 
Received on Tue Aug 24 2021 - 09:09:54 UTC

Original text of this message