From nobody Tue Feb 07 17:56:58 2023 X-Original-To: dev-commits-ports-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PB9nr6GLLz3nW6Z for ; Tue, 7 Feb 2023 17:57:36 +0000 (UTC) (envelope-from sunpoet@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PB9nr5cs5z4KQ3 for ; Tue, 7 Feb 2023 17:57:36 +0000 (UTC) (envelope-from sunpoet@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675792656; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=6AU79LNekmuZr2/IPrAV0BqqXI3LzY6a2LU+Mtmadcc=; b=aHx/hz2+40doEBL7271R263erKFEEQt/BuziKauL4D1yfYR53gugzZGYGet0lqqZFYIVbH BWs8/pCeUUbRtKbqDtJd96JFvNG0FxE+eB9sNVKiDODmWDgiXA/o415W1vD1LT55Q7ubIL kJUz8WoVhRDLXebWtehcu/m/+tZohJ6Gf+MRPPZeRWUCRiGPxsrSQ9JI+dPrc5VaeEueTq JXoqtDBRp953t8wjkIKux6Kze23gYIsvCV6QqchPkE9qDQrE1xpXbNczpnEwXYH4c4X/FD FPZ9mq4fblhQZ1KESxEr9WY/072zb5eh3xvd4ZkbyqHjPXxGrVgieWVDluMXSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675792656; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=6AU79LNekmuZr2/IPrAV0BqqXI3LzY6a2LU+Mtmadcc=; b=KnwfeFBGFB3Oiiiw2F2p3iMziMiWJLcLcAcSwp/IlLffLy6dy9RXExZVoBn2wSGWyMM4mt 7oLisw9cEX2eg0YDLzPFtITXdyiwEf4RnZh1mUDyoj7ghgNixnl5ZO3oqgZZpRDWnpGa1z fc+EhpapC4JQPJpzOeVtFewUrofQ7VZT6a6OzcxkphtendjPD33MTCuLDftacbUR7dbYoE GYQ8Nxgb//2gAbA7jtvpQGkR6tQDxC1XdUfHhuCSOJNdULa/AUsXtfHSCgT4iJYlC95D5T xVHlX1C6xCcxLKqQipt3Kp5vWuq+JsT2dsbSBIYtC8MmWVYuC49zKnvGMBJ2Iw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1675792656; a=rsa-sha256; cv=none; b=AkMG4Fp5cTYI8ecI13fwLdAAy+UCnovJVZGLzv/4y0qgf+XWnO/Z1brczkfMYRnzdb0Xsn cuPhiDDNpbG69VKlDB7JRakjQLUKXwL9+kcFx6GQVQBtD/dBYO105AgMbHY25syyMllXIe iBve7EAGwRU6KF1CJIKuhcck4/sBwP9i637poHS+znQj2Erwliaz1A1M9ttgGhaTc9g3+j +t+eAvRsulCQ/iZK+5Jqzt3ID+1Zz3PROQwcAcLryp0Q+ZaxHmiVCU9qGG8mUMlJDRay/X fp/ZkTR+f0M9eFsNC1fKHXmVR+MSFSV+3V63t86jnJSvBXnvupCqtxvlbyuOVA== Received: from mail-ed1-f52.google.com (mail-ed1-f52.google.com [209.85.208.52]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: sunpoet) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PB9nr4gtjzZBZ for ; Tue, 7 Feb 2023 17:57:36 +0000 (UTC) (envelope-from sunpoet@freebsd.org) Received: by mail-ed1-f52.google.com with SMTP id l14so7210086eds.4 for ; Tue, 07 Feb 2023 09:57:36 -0800 (PST) X-Gm-Message-State: AO0yUKXElLnLa80HnF0JfQTMnsIntKEdDANAT3+mqnRpjwd1tcPftzG8 PVEEdHk2I4J76BJwt3vZLzH9u6SPURSNqcwYdcLcpQ== X-Google-Smtp-Source: AK7set9KV+0Wb/4wj7SPvMpk0Y6uaG7QCczfrJjzJ8BxaphynmiueFqhwF3QigBrBLiXZ9vvmOBTpIdjHJW4mdJDOdo= X-Received: by 2002:a50:f689:0:b0:4a3:fc00:9d69 with SMTP id d9-20020a50f689000000b004a3fc009d69mr901376edn.6.1675792655377; Tue, 07 Feb 2023 09:57:35 -0800 (PST) List-Id: Commits to the main branch of the FreeBSD ports repository List-Archive: https://lists.freebsd.org/archives/dev-commits-ports-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-ports-main@freebsd.org X-BeenThere: dev-commits-ports-main@freebsd.org MIME-Version: 1.0 References: <202302051941.315JfQOP071383@gitrepo.freebsd.org> In-Reply-To: <202302051941.315JfQOP071383@gitrepo.freebsd.org> From: Po-Chuan Hsieh Date: Wed, 8 Feb 2023 01:56:58 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: git: f5890bd3cbc6 - main - Revert "Mk/Uses/python.mk: Fix USE_PYTHON=pep517: always compile and install bytecode" To: Charlie Li Cc: ports-committers@freebsd.org, dev-commits-ports-all@freebsd.org, dev-commits-ports-main@freebsd.org Content-Type: multipart/alternative; boundary="000000000000f1007105f41fe1d2" X-ThisMailContainsUnwantedMimeParts: N --000000000000f1007105f41fe1d2 Content-Type: text/plain; charset="UTF-8" Hello, On Mon, Feb 6, 2023 at 3:41 AM Charlie Li wrote: > The branch main has been updated by vishwin: > > URL: > https://cgit.FreeBSD.org/ports/commit/?id=f5890bd3cbc66a805bfc04a1a45688b5a3e68c50 > > commit f5890bd3cbc66a805bfc04a1a45688b5a3e68c50 > Author: Charlie Li > AuthorDate: 2023-02-05 19:05:02 +0000 > Commit: Charlie Li > CommitDate: 2023-02-05 19:05:02 +0000 > > Revert "Mk/Uses/python.mk: Fix USE_PYTHON=pep517: always compile and > install bytecode" > Despite installer's default behaviour to compile and install bytecode, > we are not doing so going forward at stage/package time. [0] During > initial development and qualification of PEP-517 framework support, > compiling and installing bytecode at stage/package time was considered, > but was found problematic, fragile and ultimately unreliable, both > currently and historically (with USE_PYTHON=distutils), due to our > fixed plist requirement. While the living binary distribution format > (wheel) specification [1] says to compile bytecode, that is in the > pure Python package management context (pip, etc); nuance always > exists when interacting with "system" package management. > > Additionally, "bytecode is an implementation detail of the CPython > interpreter. No guarantees are made that bytecode will not be added, > removed, or changed between versions of Python," thus "should not > be considered to work across Python VMs or Python releases." [2] > This is important to ensuring correctness for those ports specifying > NO_ARCH. > Those issues you mentioned might be a problem in the future but clearly not in the short term. I'm trying to fix an immediate problem which greatly affects the development. It may not be good from your point of view but it really fixed the problem (see below). If you have a better solution, I'm fine with it. > > Instead of compiling and installing bytecode at stage/package time, > there is a WIP, review D34739, that compiles and installs bytecode > at install time instead, using triggers. > Thanks for pointing out a possible solution but it is still WIP. > The aforementioned build_fs_violations will be investigated. You could reproduce this problem as follows. 1. Revert d6583d67b69dbf690e6f102e1aec799cb7fd564a (change devel/py-importlib-metadata back to USE_PYTHON=pep517 which is the background) 2. Build a port with USE_PYTHON=pep517. I choose devel/py-pyls-black here. Now it failed earlier with devel/py-click. You could expect more ports to be failed/skipped while building hundreds of ports in my case. % sudo poudriere bulk -j 12 -r -t devel/py-pyls-black [...] [00:00:10] Hit CTRL+t at any time to see build progress and stats [00:00:14] [01] [00:00:00] Building devel/py-importlib-metadata@py39 | py39-importlib-metadata-6.0.0 [00:00:19] [01] [00:00:05] Finished devel/py-importlib-metadata@py39 | py39-importlib-metadata-6.0.0: Success [00:00:19] [01] [00:00:00] Building devel/py-click@py39 | py39-click-8.1.3 [00:00:24] [01] [00:00:05] Finished devel/py-click@py39 | py39-click-8.1.3: Failed: stage_fs_violation [00:00:25] [01] [00:00:06] Skipping devel/py-black | py39-black-22.10.0_2: Dependent port devel/py-click | py39-click-8.1.3 failed [00:00:25] [01] [00:00:06] Skipping devel/py-pyls-black | py39-pyls-black-0.4.7_2: Dependent port devel/py-click | py39-click-8.1.3 failed [00:00:25] Stopping 4 builders [...] [00:01:36] Built ports: devel/py-importlib-metadata [00:01:36] Failed ports: devel/py-click:stage_fs_violation [00:01:36] Skipped ports: devel/py-black devel/py-pyls-black [...] from py39-click-8.1.3 log: [...] =>> Checking for staging violations... done =>> Error: Filesystem touched during stage (files must install to ${STAGEDIR}): extra: usr/local/lib/python3.9/site-packages/importlib_metadata/__pycache__ =>> Cleaning up wrkdir ===> Cleaning for py39-click-8.1.3 build of devel/py-click | py39-click-8.1.3 ended at Wed Feb 8 00:15:44 CST 2023 build time: 00:00:04 !!! build failure encountered !!! > This reverts commit de6965254c3a007efcf697c3d455b54d2aeb2383. > > With hat: python > Approved by: tcberner (mentor, portmgr) > Reference: https://wiki.freebsd.org/Python/PEP-517 [0] > > https://packaging.python.org/en/latest/specifications/binary-distribution-format/ > [1] > https://docs.python.org/3/library/dis.html [2] > --000000000000f1007105f41fe1d2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,

On Mon, Feb 6, 2023 at 3:41 AM Charlie Li <vishwin@freebsd.org> wrot= e:
The branch main has been updated by vishwin:

URL: https://cgi= t.FreeBSD.org/ports/commit/?id=3Df5890bd3cbc66a805bfc04a1a45688b5a3e68c50

commit f5890bd3cbc66a805bfc04a1a45688b5a3e68c50
Author:=C2=A0 =C2=A0 =C2=A0Charlie Li <vishwin@FreeBSD.org>
AuthorDate: 2023-02-05 19:05:02 +0000
Commit:=C2=A0 =C2=A0 =C2=A0Charlie Li <vishwin@FreeBSD.org>
CommitDate: 2023-02-05 19:05:02 +0000

=C2=A0 =C2=A0 Revert "Mk/Uses/python.mk: Fix USE_PYTHON=3Dpep517: always co= mpile and install bytecode"=C2=A0

=C2=A0 =C2=A0 Despite installer's default behaviour to compile and inst= all bytecode,
=C2=A0 =C2=A0 we are not doing so going forward at stage/package time. [0] = During
=C2=A0 =C2=A0 initial development and qualification of PEP-517 framework su= pport,
=C2=A0 =C2=A0 compiling and installing bytecode at stage/package time was c= onsidered,
=C2=A0 =C2=A0 but was found problematic, fragile and ultimately unreliable,= both
=C2=A0 =C2=A0 currently and historically (with USE_PYTHON=3Ddistutils), due= to our
=C2=A0 =C2=A0 fixed plist requirement. While the living binary distribution= format
=C2=A0 =C2=A0 (wheel) specification [1] says to compile bytecode, that is i= n the
=C2=A0 =C2=A0 pure Python package management context (pip, etc); nuance alw= ays
=C2=A0 =C2=A0 exists when interacting with "system" package manag= ement.

=C2=A0 =C2=A0 Additionally, "bytecode is an implementation detail of t= he CPython
=C2=A0 =C2=A0 interpreter. No guarantees are made that bytecode will not be= added,
=C2=A0 =C2=A0 removed, or changed between versions of Python," thus &q= uot;should not
=C2=A0 =C2=A0 be considered to work across Python VMs or Python releases.&q= uot; [2]
=C2=A0 =C2=A0 This is important to ensuring correctness for those ports spe= cifying
=C2=A0 =C2=A0 NO_ARCH.

Those issue= s you mentioned might be a problem in the=C2=A0future but clearly not in th= e=C2=A0short term.
I'm trying to fix an=C2=A0immediate proble= m which greatly affects the development.
It m= ay not be good from=C2=A0your point of view but it=C2=A0really fixed the pr= oblem (see below).
If you have a better solution, I&#= 39;m fine with it.
=C2=A0

=C2=A0 =C2=A0 Instead of compiling and installing bytecode at stage/package= time,
=C2=A0 =C2=A0 there is a WIP, review D34739, that compiles and installs byt= ecode
=C2=A0 =C2=A0 at install time instead, using triggers.

Thanks for pointing out a possible solution but it is still= WIP.


=C2=A0 =C2=A0 The aforementioned build_fs_violations will be investigated.<= /blockquote>

<= /div>
You could reproduce this problem as follows.
1. Revert= =C2=A0d6583d67b69dbf690e6f102e1aec799cb7fd564a (change devel/py-importlib-m= etadata back to USE_PYTHON=3Dpep517 which is the background)
2. B= uild a port with USE_PYTHON=3Dpep517. I choose devel/py-pyls-black here.
Now it failed earlier with devel/py-click.
You could expe= ct more ports to be failed/skipped while building hundreds of ports in my c= ase.

%=C2=A0sudo poudriere bulk -j 12 -r -t devel/= py-pyls-black
[...]
[00:00:10] Hit CTRL+t at any time t= o see build progress and stats
[00:00:14] [01] [00:00:00] Building devel= /py-importlib-metadata@py39 | py39-importlib-metadata-6.0.0
[00:00:19] [= 01] [00:00:05] Finished devel/py-importlib-metadata@py39 | py39-importlib-m= etadata-6.0.0: Success
[00:00:19] [01] [00:00:00] Building devel/py-clic= k@py39 | py39-click-8.1.3
[00:00:24] [01] [00:00:05] Finished devel/py-c= lick@py39 | py39-click-8.1.3: Failed: stage_fs_violation
[00:00:25] [01]= [00:00:06] Skipping devel/py-black | py39-black-22.10.0_2: Dependent port = devel/py-click | py39-click-8.1.3 failed
[00:00:25] [01] [00:00:06] Skip= ping devel/py-pyls-black | py39-pyls-black-0.4.7_2: Dependent port devel/py= -click | py39-click-8.1.3 failed
[00:00:25] Stopping 4 builders
[...]
[00:01:36] Built ports: devel/py-importlib-metadata[00:01:36] Failed ports: devel/py-click:stage_fs_violation
[00:01:36] = Skipped ports: devel/py-black devel/py-pyls-black
[...]
=

from py39-click-8.1.3 log:
[...]
= =3D>> Checking for staging violations... done
=3D>> Error: F= ilesystem touched during stage (files must install to ${STAGEDIR}):
extr= a: usr/local/lib/python3.9/site-packages/importlib_metadata/__pycache__
= =3D>> Cleaning up wrkdir
=3D=3D=3D> =C2=A0Cleaning for py39-cli= ck-8.1.3
build of devel/py-click | py39-click-8.1.3 ended at Wed Feb =C2= =A08 00:15:44 CST 2023
build time: 00:00:04
!!! build failure encount= ered !!!


=C2=A0 =C2=A0 This reverts commit de6965254c3a007efcf697c3d455b54d2aeb2383.=

=C2=A0 =C2=A0 With hat:=C2=A0 =C2=A0 =C2=A0 =C2=A0python
=C2=A0 =C2=A0 Approved by:=C2=A0 =C2=A0 tcberner (mentor, portmgr)
=C2=A0 =C2=A0 Reference:=C2=A0 =C2=A0 =C2=A0 https://wiki.fre= ebsd.org/Python/PEP-517 [0]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 https://packaging.python= .org/en/latest/specifications/binary-distribution-format/ [1]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 https://docs.python.org/3/library/dis.html [2]
--000000000000f1007105f41fe1d2--