Need help finding root cause of staging errors related to MANIFEST

[John W. O'Brien]

On 1/31/17 10:21, John W. O'Brien wrote:
> On 1/29/17 21:45, John W. O'Brien wrote:
>> At some point between 2017-01-17T07:39:59-05:00
>> 2017-01-23T10:48:15-05:00, some ports started failing to build under
>> python3.4 (not 3.5 or 3.6). The failure occurs during the poudriere
>> stage phase while processing the MANIFEST.in template. If the manifest
>> contains a graft statement that refers to a non-existent directory, the
>> build log contains a fatal error where before there was only a warning.
> [...]
>> The time interval implicates python 3.4.6, and not setuptools 32.1.0.
>> I'm having a hard time narrowing it down further than that.
> 
> After reading a bunch of source code and learning about DISTUTILS_DEBUG,
> I obtained the following trace (edited for brevity and clarity).
> 
> ====
> reading manifest template 'MANIFEST.in'
> warning: no files found matching 'README.rst'
> no previously-included directories found matching 'doc/build'
> error: [Errno 2] No such file or directory: 'examples'
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "setup.py", line 696, in <module>
>     **setuptools_kwargs)
>   File "/usr/local/lib/python3.4/distutils/core.py", line 148, in setup
>     dist.run_commands()
> [...]
>   File
> "/usr/local/lib/python3.4/site-packages/setuptools/command/egg_info.py",
> line 444, in graft
>     found = distutils.filelist.findall(dir)
>   File "/usr/local/lib/python3.4/distutils/filelist.py", line 258, in
> findall
>     names = os.listdir(dir)
> FileNotFoundError: [Errno 2] No such file or directory: 'examples'
> include MANIFEST.in
> include LICENSE
> include RELEASE.md
> include README.rst
> include setup.py
> graft doc
> prune doc/build
> graft examples
> *** Error code 1
> 
> Stop.
> make: stopped in /usr/ports/math/py-pandas
> ====
> 
> This points to a patch [0] in python 3.4.6. How does the list think I
> should proceed? File a PR with lang/python34, with devel/py-setuptools,
> with one of the upstreams?
> 
> [0] https://hg.python.org/cpython/rev/e82b995d1a5c

I compared the implementation of distutils.filelist.findall across the
four python versions supported by FreeBSD and found that 2.7 [0] and 3.4
[1] share an implementation, and 3.5 [2] and 3.6 [3] share a different
implementation. However:

$ for v in 2.7 3.4 3.5 3.6
  do
    python$v -c 'import os; os.listdir("x");' 2>&1 | tail -n1
  done
OSError: [Errno 2] No such file or directory: 'x'
FileNotFoundError: [Errno 2] No such file or directory: 'x'
FileNotFoundError: [Errno 2] No such file or directory: 'x'
FileNotFoundError: [Errno 2] No such file or directory: 'x'

So now we know that the problem only exists in 3.4 because it is the
only version (at this moment) where the findall implementation still
uses os.listdir but the os.listdir behavior for a nonexistent directory
has changed.

It's getting harder to track down A) where the change from OSError to
FileNotFoundError occurred, and B) where the OSError exception was
handled in 2.7.13 and the FileNotFoundError is not handled in 3.4.6. It
may not be necessary for the list to take a position on the bug, though.

Shall I prepare a work-around in devel/py-setuptools34, submit bug
reports to python and/or setuptools, embark upon a campaign to clean
non-existent directories from all upstream MANIFESTs, all/none of the
above, or something else?

[0]
https://hg.python.org/cpython/file/v2.7.13/Lib/distutils/filelist.py#l256
[1] https://hg.python.org/cpython/file/v3.4.6/Lib/distutils/filelist.py#l245
[2] https://hg.python.org/cpython/file/v3.5.3/Lib/distutils/filelist.py#l258
[3] https://hg.python.org/cpython/file/v3.6.0/Lib/distutils/filelist.py#l258

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-python/attachments/20170131/c6b7674d/attachment.sig>