RE: Some ports overshooting MAKE_JOBS_NUMBER

Edward Sanford Sutton, III <mirror176_at_hotmail.com> wrote on
Date: Sun, 29 Sep 2024 22:04:17 UTC :

>    Was running Poudriere with -J2 and its make.conf set to 
> MAKE_JOBS_NUMBER=6. Noticed a top terminal nearly fill with moc 
> processes and it said 30 processes were in a running state. The two 
> builds at the time were editors/calligra and graphics/digikam. Grabbing 
> part of a pstree view looks like ninja receives the -j6 parameter and 
> spawns several cmake jobs, but then each of those jobs begins spawning 
> several jobs.I know I've seen but not looked into other cases like what 
> seemed like a port's build system launching rust jobs (expected) and 
> rust launching multiple of its own jobs (unexpected) for ports like Firefox.
>    Is there active effort by porters to find and limit these cases? 
> Anything specific I should be doing to etiher further limit it or get 
> more details to track down such issues? Is there a way to have Poudriere 
> log such overuse to review without having to noice it live at the time?
> 
>    The bloated pstree copy+pasted outout is shown below:
> 
> . . .

First a MAKE_JOBS_NUMBER context-of-application note:

MAKE_JOBS_NUMBER and related settings are part of the ports
build infractucture that are involved even for direct use
of make to build ports. It is not specific to poudriere
or poudriere-devel in any way. But when poudriere runs
make, the make does what it normally does for MAKE_JOBS_NUMBER
and related settings.

So the below applies even when poudriere is not involved.


As for MAKE_JOBS_NUMBER definition:

MAKE_JOBS_NUMBER and related settings are not global constraints
on the total potential contribution to the system load average
during the make operation.

As for the 2 builders as an example (as of your
snapshot text):

"76683 root ninja -j6 -v all" of itself only directly created:
86305 86600 86968 86970 86971 86973

"81230 root ninja -j6 -v all" of itself only directly created:
86159 86876 86878 86883 86902

Anything below those in those in the process (sub)trees are
not controlled by MAKE_JOBS_NUMBER or the like and are
normally from upstream materials for building, not from
the material in the likes of /usr/ports/ .

Ports are normally designed to set up for and then use
the upstream materials for building, not for having an
independently designed and implemented build technique
to meet extra constraints that upstream does not have.

Doing more for this kind of issue could well require
complete replacements of how the upstream materials work
for building: those materials are normally designed only
for a contexts that do not have FreeBSD port's
MAKE_JOBS_NUMBER or the related settings (or other
analogous ones).

Summary: No systematic "active effort by porters to find
and limit these cases".

===
Mark Millard
marklmi at yahoo.com