Re: git: f5890bd3cbc6 - main - Revert "Mk/Uses/python.mk: Fix USE_PYTHON=pep517: always compile and install bytecode"

On Wed, Feb 8, 2023 at 2:36 AM Charlie Li <vishwin@freebsd.org> wrote:

> Po-Chuan Hsieh wrote:
> > Those issues you mentioned might be a problem in the future but clearly
> > not in the short term.
> > I'm trying to fix an immediate problem which greatly affects the
> > development.
> They are problems now, because they impede any proper fixes. We have
> been doing quite a few things wrong with Python packaging, and simply
> fixing "immediate problems" doesn't work.
> > It may not be good from your point of view but it really fixed the
> > problem (see below).
> > If you have a better solution, I'm fine with it.
> >
> >
> >          Instead of compiling and installing bytecode at stage/package
> time,
> >          there is a WIP, review D34739, that compiles and installs
> bytecode
> >          at install time instead, using triggers.
> >
> >
> > Thanks for pointing out a possible solution but it is still WIP.
> >
> It will be *the* solution, because bytecode is not meant to be packaged,
> but only compiled at install or run-time at user's discretion. The only
> reason why bytecode was packaged in operating system-level package
> distributions is how most such systems lack a trigger function that we
> now have.


> When Python packaging tooling is the final arbiter, bytecode is compiled
> only after the installation process completes if the user does not
> disable it. In our case, pkg(8) is the final arbiter, so such tasking
> falls to pkg(8)'s triggers and not during stage/packaging.
> >
> >          The aforementioned build_fs_violations will be investigated.
> >
> >
> > You could reproduce this problem as follows.
> > 1. Revert d6583d67b69dbf690e6f102e1aec799cb7fd564a (change
> > devel/py-importlib-metadata back to USE_PYTHON=pep517 which is the
> > background)
> > 2. Build a port with USE_PYTHON=pep517. I choose devel/py-pyls-black
> here.
> > Now it failed earlier with devel/py-click.
> > You could expect more ports to be failed/skipped while building hundreds
> > of ports in my case.
> >
> Can't reproduce with a proper PEP-517 bootstrap tooling, which most
> importantly does not have setuptools in the environment at all. If
> anything, your build failure only demonstrates just how "problematic,
> fragile and unreliable" compiling and installing bytecode at
> stage/package time is.
>

The problem exists.
I'm not insisting on putting bytecode in the package.
But we need a fix or hack before the solution (trigger) lands.

Let me explain it in more detail.
While building devel/py-installer, PEP517_BUILD_CMD executes
flit_core.wheel and the corresponding bytecodes are generated in the
__pycache__ directories.
The fs outside STAGEDIR is being changed.
It makes build_fs_violation failure in poudriere.
There is nothing about setuptools.

I wonder if you really tested it.
This is the up-to-date example (a fresh checkout).

% git st devel/py-flit-core devel/py-installer
On branch main
Your branch is up to date with 'freebsd/main'.

nothing to commit, working tree clean
% sudo poudriere bulk -j 12 -p testing -r -t devel/py-flit-core
devel/py-installer
[...]
[00:02:11] [01] [00:00:00] Building lang/python39 | python39-3.9.16
[00:03:13] [01] [00:01:02] Finished lang/python39 | python39-3.9.16: Success
[00:03:14] [01] [00:00:00] Building devel/py-flit-core@py39 |
py39-flit-core-3.8.0_1
[00:03:17] [01] [00:00:03] Finished devel/py-flit-core@py39 |
py39-flit-core-3.8.0_1: Success
[00:03:17] [01] [00:00:00] Building devel/py-installer@py39 |
py39-installer-0.6.0_1
[00:03:21] [01] [00:00:04] Finished devel/py-installer@py39 |
py39-installer-0.6.0_1: Failed: build_fs_violation
[...]
[00:00:44] Built ports: devel/py-flit-core
[00:00:44] Failed ports: devel/py-installer:build_fs_violation
[12-default] [2023-02-08_02h44m41s] [committing:] Queued: 2  Built: 1
 Failed: 1  Skipped: 0  Ignored: 0  Tobuild: 0   Time: 00:00:44
[...]

% tail -10 py39-installer-0.6.0_1.log
=>> Checking for filesystem violations... done
=>> Error: Filesystem touched during build:
extra:
usr/local/lib/python3.9/site-packages/flit_core/vendor/tomli/__pycache__
extra: usr/local/lib/python3.9/site-packages/flit_core/vendor/__pycache__
extra: usr/local/lib/python3.9/site-packages/flit_core/__pycache__
=>> Cleaning up wrkdir
===>  Cleaning for py39-installer-0.6.0_1
build of devel/py-installer | py39-installer-0.6.0_1 ended at Wed Feb  8
03:11:24 CST 2023
build time: 00:00:02
!!! build failure encountered !!!
% grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log
% grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log |
wc -l
       0