Re: Controlling python when building www/chromium

bob prohaska <fbsd_at_www.zefox.net> wrote on
Date: Mon, 23 Aug 2021 08:27:33 -0700 :

> When trying to build www/chromium on a 1 GB Pi3 the system
> gets bogged down by five  instances of python2.7 running
> simultaneously. This happens using both poudriere and make.
> 
> It wasn't a problem a year ago, so presumably something has
> changed in chromium's internal build machinery. I've searched
> 
> https://www.chromium.org/developers/how-tos/get-the-code
>  and
> 
> https://groups.google.com/a/chromium.org/g/chromium-dev
> 
> but couldn't find any references to python when compiling, only
> when running the browser.  
> 
> Is there some way to control how many pythons are loosed at one time?
> Most likely two could be accomodated, possibly three. On an 8 GB
> Pi4 the five pythons coexist happily, so the behavior is probably
> not considered a bug.  


Bob did not show the context. Below I show an example from his
public poudriere logs, a copy from an off-list mail, for
reference:

QUOTE
When I looked recently, the peak swap usage reported was:

Fri Aug  6 00:39:58 PDT 2021
Device          1K-blocks     Used    Avail Capacity
/dev/da0s2b       1843200  1617020   226180    88%
/dev/mmcsd0s2b    1843200  1615244   227956    88%
Total             3686400  3232264   454136    88%

and was for (showing the one after that total):

`-- /bin/sh ./buildscript.chromium
 `-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium
   |-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium
   |-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium
   `-- sh: poudriere[main-default][01]: build_pkg (chromium-91.0.4472.114_1) (sh)
     |-- sh: poudriere[main-default][01]: build_pkg (chromium-91.0.4472.114_1) (sh)
     | `-- /usr/bin/make -C /usr/ports/www/chromium build
     |   `-- (sh)
     |     `-- ninja -j1 -C out/Release chromedriver -v chrome
     |       `-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . .
     |         |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . .
     |         |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . .
     |         |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . .
     |         `-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . .
     `-- timestamp
END QUOTE

This was the most extreme swap/paging space usage from somewhat
analogous use of a generate_bindings. The swap/paging space
usage makes trying multiple builders impractical: it actually
does run out of swap/paging space. (There are limits to how
big of a swap avoids potential mistuning for a given size RAM.
swap/paging+RAM can be larger on a 8 GiByte RPi4B can be much
larger than on a RPi3B, without getting notices suggesting
a mistuned environment.)

(It is not necessarily Python 2.7. The build, overall, only uses
2.7 sometimes in some places.)

https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/blink/renderer/bindings/scripts/generate_bindings.py shows:


    bind_gen.init(web_idl_database_path=options.web_idl_database,
                  root_src_dir=options.root_src_dir,
                  root_gen_dir=options.root_gen_dir,
                  component_reldirs=component_reldirs,
                  enable_style_format=options.format_generated_files)
    task_queue = bind_gen.TaskQueue(single_process=options.single_process)
    for task in options.tasks:
        dispatch_table[task](task_queue)

which I would guess is the code initiating the parallel python
processes above.

Looking at the history, the first use of .TaskQueue here seems to have
been at:

chromium / chromium / src / 3b8f5a3b2903f2aa50efb971f430ddce57a17d16^! / . / third_party / blink / renderer / bindings / scripts / generate_bindings.py
commit	3b8f5a3b2903f2aa50efb971f430ddce57a17d16	[log] [tgz]
author	Yuki Shiino <yukishiino@chromium.org>	Thu Jun 04 05:01:43 2020
committer	Commit Bot <commit-bot@chromium.org>	Thu Jun 04 05:01:43 2020
tree	46e7ea9138d3452f1b864cbe1d9c3d1adaa62a57
parent	54c74ea75c7984bbfa11ed6232821af1943ef922 [diff] [blame]


(I did not look for other contexts with .TaskQueue uage additions.)



===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)