Re: Controlling python when building www/chromium
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Tue, 31 Aug 2021 21:05:31 UTC
From: bob prohaska <fbsd_at_www.zefox.net> wrote on Date: Tue, 24 Aug 2021 08:35:54 -0700 : > On Tue, Aug 24, 2021 at 11:09:54AM +0200, Ronald Klop wrote: > > > > Van: Mark Millard via freebsd-ports <freebsd-ports_at_freebsd.org> > > Datum: dinsdag, 24 augustus 2021 10:58 > > Aan: freebsd-ports_at_freebsd.org, bob prohaska <fbsd_at_www.zefox.net> > > Onderwerp: Re: Controlling python when building www/chromium > > > > > > > > > bob prohaska <fbsd_at_www.zefox.net> wrote on > > > Date: Mon, 23 Aug 2021 08:27:33 -0700 : > > > > > > > When trying to build www/chromium on a 1 GB Pi3 the system > > > > gets bogged down by five instances of python2.7 running > > > > simultaneously. This happens using both poudriere and make. > > > > > > > > It wasn't a problem a year ago, so presumably something has > > > > changed in chromium's internal build machinery. I've searched > > > > > > > > > https://www.chromium.org/developers/how-tos/get-the-code > > > > > and > > > > > > > > > https://groups.google.com/a/chromium.org/g/chromium-dev > > > > > > > > > but couldn't find any references to python when compiling, only > > > > when running the browser. > > > > > Is there some way to control how many pythons are loosed at one time? > > > > Most likely two could be accomodated, possibly three. On an 8 GB > > > > Pi4 the five pythons coexist happily, so the behavior is probably > > > > not considered a bug. > > > > > > > > > Bob did not show the context. Below I show an example from his > > > public poudriere logs, a copy from an off-list mail, for > > > reference: > > > > > > QUOTE > > > When I looked recently, the peak swap usage reported was: > > > > > > Fri Aug 6 00:39:58 PDT 2021 > > > Device 1K-blocks Used Avail Capacity > > > /dev/da0s2b 1843200 1617020 226180 88% > > > /dev/mmcsd0s2b 1843200 1615244 227956 88% > > > Total 3686400 3232264 454136 88% > > > > > > and was for (showing the one after that total): > > > > > > `-- /bin/sh ./buildscript.chromium > > > `-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium > > > |-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium > > > |-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium > > > `-- sh: poudriere[main-default][01]: build_pkg (chromium-91.0.4472.114_1) (sh) > > > |-- sh: poudriere[main-default][01]: build_pkg (chromium-91.0.4472.114_1) (sh) > > > | `-- /usr/bin/make -C /usr/ports/www/chromium build > > > | `-- (sh) > > > | `-- ninja -j1 -C out/Release chromedriver -v chrome > > > | `-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > > > | |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > > > | |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > > > | |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > > > | `-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > > > `-- timestamp > > > END QUOTE > > > > > > This was the most extreme swap/paging space usage from somewhat > > > analogous use of a generate_bindings. The swap/paging space > > > usage makes trying multiple builders impractical: it actually > > > does run out of swap/paging space. (There are limits to how > > > big of a swap avoids potential mistuning for a given size RAM. > > > swap/paging+RAM can be larger on a 8 GiByte RPi4B can be much > > > larger than on a RPi3B, without getting notices suggesting > > > a mistuned environment.) > > > > > > (It is not necessarily Python 2.7. The build, overall, only uses > > > 2.7 sometimes in some places.) > > > > > > > https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/blink/renderer/bindings/scripts/generate_bindings.py > shows: > > > > > > > > > bind_gen.init(web_idl_database_path=options.web_idl_database, > > > root_src_dir=options.root_src_dir, > > > root_gen_dir=options.root_gen_dir, > > > component_reldirs=component_reldirs, > > > enable_style_format=options.format_generated_files) > > > task_queue = bind_gen.TaskQueue(single_process=options.single_process) > > > for task in options.tasks: > > > dispatch_table[task](task_queue) > > > > > > which I would guess is the code initiating the parallel python > > > processes above. > > > > > > Looking at the history, the first use of .TaskQueue here seems to have > > > been at: > > > > > > chromium / chromium / src / 3b8f5a3b2903f2aa50efb971f430ddce57a17d16^! / . / third_party / blink / renderer / bindings / scripts / generate_bindings.py > > > commit 3b8f5a3b2903f2aa50efb971f430ddce57a17d16 [log] [tgz] > > > author Yuki Shiino <yukishiino_at_chromium.org> Thu Jun 04 05:01:43 2020 > > > committer Commit Bot <commit-bot_at_chromium.org> Thu Jun 04 05:01:43 2020 > > > tree 46e7ea9138d3452f1b864cbe1d9c3d1adaa62a57 > > > parent 54c74ea75c7984bbfa11ed6232821af1943ef922 [diff] [blame] > > > > > > > > > (I did not look for other contexts with .TaskQueue uage additions.) > > > > > > > > > > > > === > > > Mark Millard > > > marklmi at yahoo.com > > > ( dsl-only.net went > > > away in early 2018-Mar) > > > > > > > > > > > > > > > > > > Hi, > > > > Without looking into the code, I would guess that it should be possible to set "options.single_process" (if -j1 is set). > > As the RPI3 has 4 cores it apparently auto-scales to the amount of CPUs. > > But I don't have the time to look into the ninja build file. > > > > After the latest poudriere failure I again tried using make in > www/chromium. It also got bogged down with five pythons and > eventually stopped. At that point I simply restarted the make > session without cleaning and left the system to run overnight. > > As of this morning there are two c++ instances, the system is > only ~45% idle and it seems to be making good progress:[ 18% 5544/29872] > > The two c++ instances are consistent with /etc/make.conf . > > Does this behavior offer any hints into when and how the surplus > of pythons gets hatched? I've already shown part of the code change that runs the parallel python commands instead of one at a time --and when upstream changed that. See above. Once FreeBSD's ports updated to such a version, the port started doing the parallel subprocesses (plus the parent process). Ronald Klop already reported on what option exists to avoid such. I have looked at the code for bind_gen.TaskQueue and confirmed the option is for such. (But it is viewed as a debugging aid in how it is described in the code.) The default in the code is based on the number freebsd cpus for the number of subprocesses used (unless the option is used). (Such is in code I've not shown.) > The build session is visible via > > http://www.zefox.org/~bob/usr.ports/www/chromium/ No, when I looked at the time, the only log file was from the 2nd run after the problem occurred and so only had information from you started tat 2nd make run. Note: At the time a fair amount of old material was still exposed from prior poudriere experiments and such. If they are still exposed, you might want to remove them. > The actvivity logging script is still running from the poudriere > session and the output is visible at > > http://www.zefox.org/~bob/swaplogs/20210816.log > > however the file is now somewhat unwieldy in size. Note: it might help to stop this and start another one with a new file once and a while (daily?). Such would make getting information of interst more reasonable. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)