Re: Controlling python when building www/chromium
- In reply to: Ronald Klop : "Re: Controlling python when building www/chromium"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Tue, 24 Aug 2021 15:35:54 UTC
On Tue, Aug 24, 2021 at 11:09:54AM +0200, Ronald Klop wrote: > > Van: Mark Millard via freebsd-ports <freebsd-ports@freebsd.org> > Datum: dinsdag, 24 augustus 2021 10:58 > Aan: freebsd-ports@freebsd.org, bob prohaska <fbsd@www.zefox.net> > Onderwerp: Re: Controlling python when building www/chromium > > > > > > bob prohaska <fbsd_at_www.zefox.net> wrote on > > Date: Mon, 23 Aug 2021 08:27:33 -0700 : > > > > > When trying to build www/chromium on a 1 GB Pi3 the system > > > gets bogged down by five instances of python2.7 running > > > simultaneously. This happens using both poudriere and make. > > > > > > It wasn't a problem a year ago, so presumably something has > > > changed in chromium's internal build machinery. I've searched > > > > > > https://www.chromium.org/developers/how-tos/get-the-code > > > and > > > > > > https://groups.google.com/a/chromium.org/g/chromium-dev > > > > > > but couldn't find any references to python when compiling, only > > > when running the browser. > > > > Is there some way to control how many pythons are loosed at one time? > > > Most likely two could be accomodated, possibly three. On an 8 GB > > > Pi4 the five pythons coexist happily, so the behavior is probably > > > not considered a bug. > > > > > > Bob did not show the context. Below I show an example from his > > public poudriere logs, a copy from an off-list mail, for > > reference: > > > > QUOTE > > When I looked recently, the peak swap usage reported was: > > > > Fri Aug 6 00:39:58 PDT 2021 > > Device 1K-blocks Used Avail Capacity > > /dev/da0s2b 1843200 1617020 226180 88% > > /dev/mmcsd0s2b 1843200 1615244 227956 88% > > Total 3686400 3232264 454136 88% > > > > and was for (showing the one after that total): > > > > `-- /bin/sh ./buildscript.chromium > > `-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium > > |-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium > > |-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium > > `-- sh: poudriere[main-default][01]: build_pkg (chromium-91.0.4472.114_1) (sh) > > |-- sh: poudriere[main-default][01]: build_pkg (chromium-91.0.4472.114_1) (sh) > > | `-- /usr/bin/make -C /usr/ports/www/chromium build > > | `-- (sh) > > | `-- ninja -j1 -C out/Release chromedriver -v chrome > > | `-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > > | |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > > | |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > > | |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > > | `-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > > `-- timestamp > > END QUOTE > > > > This was the most extreme swap/paging space usage from somewhat > > analogous use of a generate_bindings. The swap/paging space > > usage makes trying multiple builders impractical: it actually > > does run out of swap/paging space. (There are limits to how > > big of a swap avoids potential mistuning for a given size RAM. > > swap/paging+RAM can be larger on a 8 GiByte RPi4B can be much > > larger than on a RPi3B, without getting notices suggesting > > a mistuned environment.) > > > > (It is not necessarily Python 2.7. The build, overall, only uses > > 2.7 sometimes in some places.) > > > > https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/blink/renderer/bindings/scripts/generate_bindings.py shows: > > > > > > bind_gen.init(web_idl_database_path=options.web_idl_database, > > root_src_dir=options.root_src_dir, > > root_gen_dir=options.root_gen_dir, > > component_reldirs=component_reldirs, > > enable_style_format=options.format_generated_files) > > task_queue = bind_gen.TaskQueue(single_process=options.single_process) > > for task in options.tasks: > > dispatch_table[task](task_queue) > > > > which I would guess is the code initiating the parallel python > > processes above. > > > > Looking at the history, the first use of .TaskQueue here seems to have > > been at: > > > > chromium / chromium / src / 3b8f5a3b2903f2aa50efb971f430ddce57a17d16^! / . / third_party / blink / renderer / bindings / scripts / generate_bindings.py > > commit 3b8f5a3b2903f2aa50efb971f430ddce57a17d16 [log] [tgz] > > author Yuki Shiino <yukishiino@chromium.org> Thu Jun 04 05:01:43 2020 > > committer Commit Bot <commit-bot@chromium.org> Thu Jun 04 05:01:43 2020 > > tree 46e7ea9138d3452f1b864cbe1d9c3d1adaa62a57 > > parent 54c74ea75c7984bbfa11ed6232821af1943ef922 [diff] [blame] > > > > > > (I did not look for other contexts with .TaskQueue uage additions.) > > > > > > > > === > > Mark Millard > > marklmi at yahoo.com > > ( dsl-only.net went > > away in early 2018-Mar) > > > > > > > > > > > Hi, > > Without looking into the code, I would guess that it should be possible to set "options.single_process" (if -j1 is set). > As the RPI3 has 4 cores it apparently auto-scales to the amount of CPUs. > But I don't have the time to look into the ninja build file. > After the latest poudriere failure I again tried using make in www/chromium. It also got bogged down with five pythons and eventually stopped. At that point I simply restarted the make session without cleaning and left the system to run overnight. As of this morning there are two c++ instances, the system is only ~45% idle and it seems to be making good progress:[ 18% 5544/29872] The two c++ instances are consistent with /etc/make.conf . Does this behavior offer any hints into when and how the surplus of pythons gets hatched? The build session is visible via http://www.zefox.org/~bob/usr.ports/www/chromium/ The actvivity logging script is still running from the poudriere session and the output is visible at http://www.zefox.org/~bob/swaplogs/20210816.log however the file is now somewhat unwieldy in size. Thanks for reading, and any thoughts. bob prohaska