python waf bypasses _MAKE_JOBS number

[Sean Bruno]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 01/07/15 10:51, sson [via FreeBSD] wrote:
> Sean Bruno-6 wrote
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
>> 
>> On 01/07/15 00:21, Marcus von Appen wrote:
>>> On, Wed Jan 07, 2015, Sean Bruno wrote:
>>> 
>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
>>>> 
>>>> Hey, so ... because qemu has a bug in it, we're trying to
>>>> enforce no SMP behaviour in builds.
>>>> 
>>>> Turns out that python waf bypasses all of ports logic and
>>>> probes for the number of cpus and does its own thing.  We
>>>> noted this in our builds as they locked up when using qemu
>>>> due to a bug.
>>>> 
>>>> Can this behaviour be investigated (configure behaviour) by
>>>> some python knowledgeable folks?
>>> 
>>> Can you point us to the waf build logic for the qemu port?
>>> Looking at emulators/qemu, I do not see anything of relevance.
>>> 
>>> Cheers Marcus
>>> 
>> 
>> Should have been a bit clearer, its qemu-bsd-user (via 
>> qemu-user-static) that I'm talking about.
>> 
>> I've added Stacey to the email to clarify a bit more.
>> 
>> The behavior we are talking about isn't specific to qemu at all,
>> its just the configure script will probe the number of cpus and
>> ignore my override.
>> 
>> sean _______________________________________________
> 
>> freebsd-python@
> 
>> mailing list 
>> http://lists.freebsd.org/mailman/listinfo/freebsd-python To
>> unsubscribe, send any mail to "
> 
>> freebsd-python-unsubscribe@
> 
>> "
> 
> Hi all:
> 
> The problem we are seeing is actually described in pretty good
> detail in an old a bug report:
> 
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=160717
> 
> It is hard to follow what was actually changed that fixed this
> problem on FreeBSD/amd64 but it seems the threading code for
> FreeBSD/{arm, mips, mips64} may not be as mature as the amd64 port.
> In any case, if the "--jobs=1" flag is used then the ports will
> build without hanging.  Python waf assumes, however, that the
> default jobs number should be the number of cores available on the
> system when the "--jobs=#" (or "-j#") is not explicitly given
> rather than assuming "--jobs=1".   It seems, however, that many (if
> not all) the ports that use python waf make the assumption that it 
> defaults to "--jobs=1". So, in summary:
> 
> (1) It seems that described bug above wasn't fixed for all the
> FreeBSD arch's.  It is unclear was the cause of the problem was and
> what was changed.  If it is know what was changed it may be
> possible to figure out what needs to be changed to fix this problem
> in the other FreeBSD ports.
> 
> (2) It seems to be assumed by many of the port maintainers that
> python waf defaults to "--jobs=1" when the argument is not given.
> This is problematic for poudriere, for example, given that it
> limits the MAKE_JOBS_NUMBER to 1 to better control the load on the
> build server.
> 
> Of course, if #2 was fixed then it would solve two problems (for us
> anyway). :)
> 
> Best Regards,
> 
> -stacey.
> 
> 
> 

I think I can work around python waf assuming it can figure out the
number of cpus in the two critical ports tdb/talloc by enforcing a
- --jobs=1

https://people.freebsd.org/~sbruno/disable_make_jobs.diff

These two are the most obvious in the tree that fail for armv6 qemu
builds.

sean

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQF8BAEBCgBmBQJUraRkXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx
MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5kg6kIAMpyFOqqIxhbsDI1HBFAWIM6
j/ci3S1i7H0uVzoTmvGn3ITDY8zAPG3cDm1EaPPcn5vUNvAnk9uzo0ip33OZ22WM
8mJza2vGyoWLFGdrZPwm/+izWdXDoGqLX1ZSvzQlT8elyD2DlzduX++UJNdLgBkU
assRj4HUFxe/1OQO911bZXfQVrsU9XbvssDrCvgZlQGo7siEDrQUpbQgqfDviQjK
SXxJy8qM7l60309w13TmZpCriE3nTmNMdUazTV5mXS/NhQqkL5ckiv0uHGrn6/a7
KC/L+whcjuT4FF86n0qCyrBiSsEemxOywIIdVd+r2X75r0pmXCw6214CI6QuZ40=
=f2RT
-----END PGP SIGNATURE-----