Re: poudriere-devel bulk -a python-related www/chromium build failures (aarch64 context)

On 2021-Sep-23, at 15:07, Mark Millard <marklmi@yahoo.com> wrote:

> So far my attempts at an aarch64 native bulk -a have had
> www/chromium fail with:
> 
> . . .
> python3 ../../build/util/python2_action.py ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle --r
> oot_src_dir ../../ --root_gen_dir gen --output_core_reldir third_party/blink/renderer/bindings/core/v8/ --output_modules_reldir third_party/blink/renderer/bindings/modules/v8/ enumeration callback_fun
> ction callback_interface interface namespace typedef union
> Traceback (most recent call last):
>  File "/wrkdirs/usr/ports/www/chromium/work/chromium-92.0.4515.159/buildtools/freebsd/clang-format", line 34, in <module>
>    stdout, stderr = proc.communicate(input=contents)
>  File "/usr/local/lib/python3.8/subprocess.py", line 1028, in communicate
>    stdout, stderr = self._communicate(input, endtime, timeout)
>  File "/usr/local/lib/python3.8/subprocess.py", line 1848, in _communicate
>    input_view = memoryview(self._input)
> TypeError: memoryview: a bytes-like object is required, not 'str'
> Traceback (most recent call last):
>  File "/wrkdirs/usr/ports/www/chromium/work/chromium-92.0.4515.159/buildtools/freebsd/clang-format", line 34, in <module>
>    stdout, stderr = proc.communicate(input=contents)
>  File "/usr/local/lib/python3.8/subprocess.py", line 1028, in communicate
>    stdout, stderr = self._communicate(input, endtime, timeout)
>  File "/usr/local/lib/python3.8/subprocess.py", line 1848, in _communicate
>    input_view = memoryview(self._input)
> TypeError: memoryview: a bytes-like object is required, not 'str'
> . . . (more such tracebacks) . . .
> 
> I do not see such a failure in the likes of:
> 
> http://ampere2.nyi.freebsd.org/data/main-arm64-default/p5f5c0fe00a09_s3fcbde5e88/logs/chromium-92.0.4515.159_2.log
> 
> I've yet to figure out why my build attempts are different.
> 
> For reference:
> 
> The deafult lvm is llvm12.
> 
> # pwd
> /usr/ports
> # ~/fbsd-based-on-what-commit.sh 
> branch: main
> merge-base: a59b4fa83271ce1a7610973b2a5f3f0362e4bfba
> merge-base: CommitDate: 2021-09-21 01:12:16 +0000
> a59b4fa83271 (HEAD -> main, freebsd/main, freebsd/HEAD) security/py-scramp: Update to 1.4.1
> n558469 (--first-parent --count for merge-base)
> 
> # uname -apKU
> FreeBSD CA72_16Gp_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #12 main-n249019-0637070b5bca-dirty: Tue Aug 31 02:24:20 PDT 2021     root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA72  arm64 aarch64 1400032 1400032
> 
> I do not normally build chromium or do bulk -a runs.
> My activity is exploratory to see what problems I'd
> run into and if I'd need adjustments to do bulk -a
> runs successfully. (It has lead to the (root) file
> system being made larger, for example. I also disabled
> lang/ratfor because the port does not deal with
> platforms that have char as unsigned but the port is
> coded requiring a form of signed char. This avoided
> unbounded looping and a timeout.)

When I look at:

/wrkdirs/usr/ports/www/chromium/work/chromium-92.0.4515.159/buildtools/freebsd/clang-format

I see (in part):

. . .
contents = ''
if '-' in args or not inputfiles:
    contents = sys.stdin.read()
. . .
    proc = subprocess.Popen(
        ['clang-format'] + args,
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        env=env)
    stdout, stderr = proc.communicate(input=contents)
. . .

Which looks to me to be passing a str, not a bytes or bytearray
and not None.

Also Popen.stdin is documented with:

QUOTE
Popen.stdin
If the stdin argument was PIPE, this attribute is a writeable stream object
as returned by open(). If the encoding or errors arguments were specified
or the universal_newlines argument was True, the stream is a text stream,
otherwise it is a byte stream. If the stdin argument was not PIPE, this
attribute is None.
END QUOTE

So, if I gather correctly, stdin ends up being a byte stream acording
to the documenation.

proc.communicate is documented with "If streams were opened in text
mode, input must be a string. Otherwise, it must be bytes." as well.

And that in turn involves:

/usr/local/lib/python3.8/subprocess.py has:

    def communicate(self, input=None, timeout=None):
. . .
                stdout, stderr = self._communicate(input, endtime, timeout)
. . .

So inpput still seems to be a str for the following:

        def _communicate(self, input, endtime, orig_timeout):
. . .
            self._save_input(input)            
            if self._input:
                input_view = memoryview(self._input)
. . .

but self._input need not be.

This is where:

        def _save_input(self, input):
            # This method is called from the _communicate_with_*() methods
            # so that if we time out while communicating, we can continue
            # sending input if we retry.
            if self.stdin and self._input is None:
                self._input_offset = 0
                self._input = input
                if input is not None and self.text_mode:
                    self._input = self._input.encode(self.stdin.encoding,
                                                     self.stdin.errors)

The implication of the error would seem to be
that the encode did not happen and, so, either

self.stdin and self._input is None

was false or:

self.text_mode

was false (and it appears to be the later).

This fits with the stdin being a byte stream but input being a str
--and so the error report from memoryview.

But I still have no clue why the FreeBSD server builds of chromium
do not end up with such an error report.

Note: I've duplicated the failure in my amd64 context. The issue is
not aarch64 specific in my environment.


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)