Re: git: f5890bd3cbc6 - main - Revert "Mk/Uses/python.mk: Fix USE_PYTHON=pep517: always compile and install bytecode"

Hello,

On Mon, Feb 6, 2023 at 3:41 AM Charlie Li <vishwin@freebsd.org> wrote:

> The branch main has been updated by vishwin:
>
> URL:
> https://cgit.FreeBSD.org/ports/commit/?id=f5890bd3cbc66a805bfc04a1a45688b5a3e68c50
>
> commit f5890bd3cbc66a805bfc04a1a45688b5a3e68c50
> Author:     Charlie Li <vishwin@FreeBSD.org>
> AuthorDate: 2023-02-05 19:05:02 +0000
> Commit:     Charlie Li <vishwin@FreeBSD.org>
> CommitDate: 2023-02-05 19:05:02 +0000
>
>     Revert "Mk/Uses/python.mk: Fix USE_PYTHON=pep517: always compile and
> install bytecode"


>     Despite installer's default behaviour to compile and install bytecode,
>     we are not doing so going forward at stage/package time. [0] During
>     initial development and qualification of PEP-517 framework support,
>     compiling and installing bytecode at stage/package time was considered,
>     but was found problematic, fragile and ultimately unreliable, both
>     currently and historically (with USE_PYTHON=distutils), due to our
>     fixed plist requirement. While the living binary distribution format
>     (wheel) specification [1] says to compile bytecode, that is in the
>     pure Python package management context (pip, etc); nuance always
>     exists when interacting with "system" package management.
>
>     Additionally, "bytecode is an implementation detail of the CPython
>     interpreter. No guarantees are made that bytecode will not be added,
>     removed, or changed between versions of Python," thus "should not
>     be considered to work across Python VMs or Python releases." [2]
>     This is important to ensuring correctness for those ports specifying
>     NO_ARCH.
>

Those issues you mentioned might be a problem in the future but clearly not
in the short term.
I'm trying to fix an immediate problem which greatly affects the
development.
It may not be good from your point of view but it really fixed the problem
(see below).
If you have a better solution, I'm fine with it.


>
>     Instead of compiling and installing bytecode at stage/package time,
>     there is a WIP, review D34739, that compiles and installs bytecode
>     at install time instead, using triggers.
>

Thanks for pointing out a possible solution but it is still WIP.


>     The aforementioned build_fs_violations will be investigated.


You could reproduce this problem as follows.
1. Revert d6583d67b69dbf690e6f102e1aec799cb7fd564a (change
devel/py-importlib-metadata back to USE_PYTHON=pep517 which is the
background)
2. Build a port with USE_PYTHON=pep517. I choose devel/py-pyls-black here.
Now it failed earlier with devel/py-click.
You could expect more ports to be failed/skipped while building hundreds of
ports in my case.

% sudo poudriere bulk -j 12 -r -t devel/py-pyls-black
[...]
[00:00:10] Hit CTRL+t at any time to see build progress and stats
[00:00:14] [01] [00:00:00] Building devel/py-importlib-metadata@py39 |
py39-importlib-metadata-6.0.0
[00:00:19] [01] [00:00:05] Finished devel/py-importlib-metadata@py39 |
py39-importlib-metadata-6.0.0: Success
[00:00:19] [01] [00:00:00] Building devel/py-click@py39 | py39-click-8.1.3
[00:00:24] [01] [00:00:05] Finished devel/py-click@py39 | py39-click-8.1.3:
Failed: stage_fs_violation
[00:00:25] [01] [00:00:06] Skipping devel/py-black | py39-black-22.10.0_2:
Dependent port devel/py-click | py39-click-8.1.3 failed
[00:00:25] [01] [00:00:06] Skipping devel/py-pyls-black |
py39-pyls-black-0.4.7_2: Dependent port devel/py-click | py39-click-8.1.3
failed
[00:00:25] Stopping 4 builders
[...]
[00:01:36] Built ports: devel/py-importlib-metadata
[00:01:36] Failed ports: devel/py-click:stage_fs_violation
[00:01:36] Skipped ports: devel/py-black devel/py-pyls-black
[...]

from py39-click-8.1.3 log:
[...]
=>> Checking for staging violations... done
=>> Error: Filesystem touched during stage (files must install to
${STAGEDIR}):
extra: usr/local/lib/python3.9/site-packages/importlib_metadata/__pycache__
=>> Cleaning up wrkdir
===>  Cleaning for py39-click-8.1.3
build of devel/py-click | py39-click-8.1.3 ended at Wed Feb  8 00:15:44 CST
2023
build time: 00:00:04
!!! build failure encountered !!!


>     This reverts commit de6965254c3a007efcf697c3d455b54d2aeb2383.
>
>     With hat:       python
>     Approved by:    tcberner (mentor, portmgr)
>     Reference:      https://wiki.freebsd.org/Python/PEP-517 [0]
>
> https://packaging.python.org/en/latest/specifications/binary-distribution-format/
> [1]
>                     https://docs.python.org/3/library/dis.html [2]
>