From nobody Fri Jan 26 08:01:17 2024 X-Original-To: ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TLqsR1tZ0z57qjm for ; Fri, 26 Jan 2024 08:01:47 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TLqsQ2RRVz4fQB; Fri, 26 Jan 2024 08:01:46 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; none List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1706256098; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=h4sxcBSjok47Mt3NiPcUljDMRVdaeNMs4OjxasdpurY=; b=F306yJgW2jsP7YRopSlIYQj79WnkZqLIjvGbXOi2w0tVrkcUcDscln4UFoj6WEbYOKhNmA GRK3WuTxfUgq+enf0XkvCj/93I5XBpzCSoBesC2Y/7t7uUCFQd5yr9V3bExrG9rF4EP5je BJQcaknYozF899o/pr76sxv5okE+exsliGeWlHJXAzY0VuIyDnS6jcXrx++7GpCMhSCV7W y4fI5ly+mtSnP00fIUGsJtKswaF5DmZxYEzOefHQprAES3ylS8icVN40gmsi1kFCflVrP0 Mc9K7RwJf+Nl0MChl2ncDXCZkmN1/SWJ2vaxHAuwMuEy6rp9YDlFU8VSbOAp2A== Date: Fri, 26 Jan 2024 09:01:17 +0100 From: Alexander Leidinger To: Luca Pizzamiglio Cc: Stefan Esser , freebsd-ports , portmgr , FreeBSD Core Team Subject: Re: This is going to break port building without poudriere! In-Reply-To: References: Message-ID: Organization: No organization, this is a private message. Content-Type: multipart/signed; protocol="application/pgp-signature"; boundary="=_afcc090eeb35f4cae43fa1bcd152af2b"; micalg=pgp-sha256 X-Rspamd-Queue-Id: 4TLqsQ2RRVz4fQB X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE] This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --=_afcc090eeb35f4cae43fa1bcd152af2b Content-Type: multipart/alternative; boundary="=_9bd8e07784132f49b91be5cdcb1f51e3" --=_9bd8e07784132f49b91be5cdcb1f51e3 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; format=flowed Am 2024-01-25 20:57, schrieb Luca Pizzamiglio: > Hi Stefan, > > I did reply to your first email, but not to your second one. > I preferred (in agreement with portmgr@) to open the discussion with > everyone, instead of keeping it just between you and me. > > As you can read in the email you have linked, I didn't ignore your > comments. > >> This implementation will break port dependencies, since there is no >> way >> a port can depend on a specific sub-package - there even is no way a >> non-default sub-package can be built without manual selection of the >> options that activate its creation. > > A port can depend on a specific sub-package. The category/origin~subpkg > is the chosen format. > Non-default subpackages shouldn't exist. In the aforementioned email, I > explicitly say that IF a subpackage is enabled by an option, the option > must be enabled by default. > Your previous comments contributed to making this point clearer. > >> Dependencies stated in the port Makefile are converted into package >> dependencies that can be resolved by the "pkg" command, but that >> cannot >> be directly used to build and install the requested sub-package from >> a port. > > I don't know where you get this idea, but it's not how it works. > If a port has a subpackage dependency, the category/origin~subpkg is > the chosen format to express that dependency. > The related port (category/origin) has to be built and installed. > By running `make install`, the whole port is installed, subpackages as > well. > The only "issue" I see is that via 'make install' you cannot install > only the subpackage, but the entire port only. > However this is not different than before. I think Stefans expectations about such a feature are different from the understand of the implementor of the feature in our tree. Somewhat a clash of an idealistic view and reality. To me (and I assume to Stefan too) it looks like slave ports will go away and subpackages will be used instead. A slave port may only compile a subset, and a subpackages aware port will compile everything (simplified view, not true where slave ports exclude a feature instead of excluding a file). From this point of view, a port can not depend on a subpackage in the sense of a port can not depend only on a subset of the php extensions included in the main php build (if/when it is converted to subpackages). As such a build from ports will have all php extensions included in the php port installed, whereas a pkp based install can limit the amount of installed php extensions to what is required. At least this is what I understand based upon what I have read about subpackages. As the documentation is not ready and I haven't looked at the code, this understanding may off course be wrong. >> Ports that do not create sub-packages can still be depended on by >> other >> ports, but as critical dependencies have been depended to sub-packages >> a ever large fraction of ports will only build in poudriere. > > Again, no. > If you run make install on any port, it will be built. The package and > the subpackages are all installed. > You cannot install a portion of a port, and it has never been possible. Is my understanding correct that subpackages aware ports are supposed to replace master/slave ports? In the sense of slave ports will be deleted once a port is converted to a subpackages aware port (except where slave ports exclude features from binaries like git-lite... yes, git-lite is implemented as a flavour and as such we cover this in a different way and such slave ports if they still exist can maybe be converted to use flavours)? I assume yes to both questions. With this assumption: With master/slave ports this is possible. And replacing master/slave ports with a subpackages aware port will remove this possibility. > Nothing is broken here and I fail to understand where you get the idea > of this behavior. > I wrote tests and examples to implement the feature and get them > working before adding support to poudriere. > >> This change does also obviously break port management tools like >> portmaster, >> which took me significant effort to adapt to FLAVOR support (which >> also had >> been implemented without consideration for other tools than >> poudriere), and >> which I have been maintaining since then. > > Yes, portmaster need to be able to parse the new dependencies format, > by removing the ~subpkg. By removing the suffix, you get the > category/port and everything is fine. > As announced, we are keeping subpackages adoption blocked with a > git-hook to give time to maintainers to add support to subpkg and to > introduce the feature slowly. > portmgr@ doesn't officially support neither portmaster nor portupgrade. > We simply lack the manpower to do it. > >> The reason given for sub-packages support is reduced build time for >> some >> packages that share a common distfile (e.g. qt5) when building >> official >> packages. > > As explained, in the short term there are this benefit and getting rid > of many master/slave ports (hard to maintain) > But there are several use cases in the future. > For instance, work is in place to provide debug symbols as subpackage, > in a similar way as pkg-base. > >> But this comes at a high cost for all builds outside the package build >> cluster, since now lots of unnecessary sub-trees will be compiled and >> installed, if only one program (i.e. sub-package) is desired. > > This is the reason why OPTIONS with SUBPACAKGES have been introduced, > to reduce build time for port builders. The complaint/fear of Stefan is that there does not seem to be a way to e.g.: 1) have php converted to a subpackages aware port which will remove some of the php extensions ports as they will be handled as subpackages of the lang/php port 2) depend upon one of the extensions which are now integrated as a subpackage of the php port but do not depend upon all extensions which are available as subpackages 3) as a result have all php extensions build and installed (and as such loaded at run time automatically), even those which are not needed, when installing via ports I think what Stefan is asking for, is to have a way that only build a subpackage if desired (default behavior = build everything, but the possibility to only build the subpackage specific part in case the upstream source allows this, like in the php example). >> You are breaking use-cases of a large number of users that still build >> ports without poudriere. This is especially causing a barrier to entry >> of new port developers, since this will require them to setup >> poudriere >> before they can begin port development. (This will only become an >> issue >> over time, as more and more dependencies will have been converted to >> sub-packages, but then there will be no way back.) > > I still fail to see what is going to break. Non-poudriere users can > experience longer build time for dependencies, but dependencies can be > installed via packages, at least most of the time. What is broken is that you _have_ to install such dependencies via packages in this case if you want to have the minimal install. With master/slave ports this is not required, the user is free to install everything as a port. "cd /usr/ports/port/which/depends/upon/a/subset/of/the/php/extensions; make install" will be different than "pkg install port/which/depends/upon/a/subset/of/the/php/extensions". And in this example this can also have a security implication (php stuff may be exposed to the internet and more extensions which can have a security issue may as such be exposed when installed as a port but not when installed as a package). Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_9bd8e07784132f49b91be5cdcb1f51e3 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=UTF-8

Am 2024-01-25 20:57, schrieb Luca Pizzamiglio:

Hi Stefan,
 
I did reply to your first email, but not to your second one.
I pr= eferred (in agreement with portmgr@) to open the discussion with everyone, = instead of keeping it just between you and me.
 
As you can read in the email you have linked, I didn't ignore your com= ments.
 
This implementation will b= reak port dependencies, since there is no way
a port can depend on a s= pecific sub-package - there even is no way a
non-default sub-package c= an be built without manual selection of the
options that activate its = creation.
A port can depend on a specific sub-package. The category/origin~subpk= g is the chosen format.
Non-default subpackages shouldn't exist. In the aforementioned email, = I explicitly say that IF a subpackage is enabled by an option, the option m= ust be enabled by default.
Your previous comments contributed to makin= g this point clearer.
 
Dependencies stated in the= port Makefile are converted into package
dependencies that can be res= olved by the "pkg" command, but that cannot
be directly used to build = and install the requested sub-package from
a port.
I don't know where you get this idea, but it's not how it works.
If a port has a subpackage dependency, the category/origin~subpkg is t= he chosen format to express that dependency.
The related port (categor= y/origin) has to be built and installed.
By running `make install`, the whole port is installed, subpackages as= well.
The only "issue" I see is that via 'make install' you cannot install o= nly the subpackage, but the entire port only.
However this is not different than before.
 
I think Stefans expectations about such a feature are different from t= he understand of the implementor of the feature in our tree. Somewhat a cla= sh of an idealistic view and reality.
 
To me (and I assume to Stefan too) it looks like slave ports will go a= way and subpackages will be used instead. A slave port may only compile a s= ubset, and a subpackages aware port will compile everything (simplified vie= w, not true where slave ports exclude a feature instead of excluding a file= ). From this point of view, a port can not depend on a subpackage in the se= nse of a port can not depend only on a subset of the php extensions include= d in the main php build (if/when it is converted to subpackages). As such a= build from ports will have all php extensions included in the php port ins= talled, whereas a pkp based install can limit the amount of installed php e= xtensions to what is required. At least this is what I understand based upo= n what I have read about subpackages. As the documentation is not ready and= I haven't looked at the code, this understanding may off course be wrong.<= /div>
 
Ports that do not create s= ub-packages can still be depended on by other
ports, but as critical d= ependencies have been depended to sub-packages
a ever large fraction o= f ports will only build in poudriere.
Again, no.
If you run make install on any port, it will be built. The package and= the subpackages are all installed.
You cannot install a portion of a port, and it has never been possible= =2E
 
Is my understanding correct that subpackages aware ports are supposed = to replace master/slave ports? In the sense of slave ports will be deleted = once a port is converted to a subpackages aware port (except where slave po= rts exclude features from binaries like git-lite... yes, git-lite is implem= ented as a flavour and as such we cover this in a different way and such sl= ave ports if they still exist can maybe be converted to use flavours)? I as= sume yes to both questions.  With this assumption:
 
With master/slave ports this is possible. And replacing master/slave p= orts with a subpackages aware port will remove this possibility.
 
Nothing is broken here and I fail to understand where you get the idea= of this behavior.
I wrote tests and examples to implement the feature and get them worki= ng before adding support to poudriere.
 
This change does also obvi= ously break port management tools like portmaster,
which took me signi= ficant effort to adapt to FLAVOR support (which also had
been implemen= ted without consideration for other tools than poudriere), and
which I= have been maintaining since then.
Yes, portmaster need to be able to parse the new dependencies format, = by removing the ~subpkg. By removing the suffix, you get the category/port = and everything is fine.
As announced, we are keeping subpackages adoption blocked with a git-h= ook to give time to maintainers to add support to subpkg and to introduce t= he feature slowly.
portmgr@ doesn't officially support neither portmaster nor portupgrade= =2E We simply lack the manpower to do it.
 
The reason given for sub-p= ackages support is reduced build time for some
packages that share a c= ommon distfile (e.g. qt5) when building official
packages.
As explained, in the short term there are this benefit and getting rid= of many master/slave ports (hard to maintain)
But there are several use cases in the future.
For instance, work is in place to provide debug symbols as subpackage,= in a similar way as pkg-base.
 
But this comes at a high c= ost for all builds outside the package build
cluster, since now lots o= f unnecessary sub-trees will be compiled and
installed, if only one pr= ogram (i.e. sub-package) is desired.
This is the reason why OPTIONS with SUBPACAKGES have been introduced, = to reduce build time for port builders.
 
The complaint/fear of Stefan is that there does not seem to be a way t= o e.g.:
 1) have php converted to a subpackages aware port which will rem= ove some of the php extensions ports as they will be handled as subpackages= of the lang/php port
 2) depend upon one of the extensions which are now integrated as= a subpackage of the php port but do not depend upon all extensions which a= re available as subpackages
 3) as a result have all php extensions build and installed (and = as such loaded at run time automatically), even those which are not needed,= when installing via ports
 
I think what Stefan is asking for, is to have a way that only build a = subpackage if desired (default behavior =3D build everything, but the possi= bility to only build the subpackage specific part in case the upstream sour= ce allows this, like in the php example).
 
You are breaking use-cases= of a large number of users that still build
ports without poudriere. = This is especially causing a barrier to entry
of new port developers, = since this will require them to setup poudriere
before they can begin = port development. (This will only become an issue
over time, as more a= nd more dependencies will have been converted to
sub-packages, but the= n there will be no way back.)
I still fail to see what is going to break.
Non-poudriere users can experience longer build time for dependencies, but = dependencies can be installed via packages, at least most of the time.
 
What is broken is that you _have_ to install s= uch dependencies via packages in this case if you want to have the minimal = install. With master/slave ports this is not required, the user is free to = install everything as a port. "cd /usr/ports/port/which/depends/upon/a/subs= et/of/the/php/extensions; make install" will be different than "pkg install= port/which/depends/upon/a/subset/of/the/php/extensions". And in this examp= le this can also have a security implication (php stuff may be exposed to t= he internet and more extensions which can have a security issue may as such= be exposed when installed as a port but not when installed as a package).<= /div>
 
Bye,
Alexander.
 
--
= http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF
http:/= /www.FreeBSD.org    n= etchild@FreeBSD.org  : PGP 0x8F31830F9F2772BF
--=_9bd8e07784132f49b91be5cdcb1f51e3-- --=_afcc090eeb35f4cae43fa1bcd152af2b Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc; size=833 Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmWzZt4ACgkQEg2wmwP4 2IZqIQ//S2Wka4A3HjAcX5aEPJ++eST8tyvs6Eq+4wXp1oumgL7378cilqNG5NMB QZmi9CoX3OxTJhbmL/HM6iKCc79Nu2cIuBMk6AbaGWG3VzeZIYTpOVQJUtJWgHD1 8TelAR+iiwb7fe2hUjEqaYekYl4bJu+oQxuX0GpPFFaw0UQJp7Fccw2KxISHe/L9 MOMsdqjvh0qj05Lmr94a14xdzAKWW6g5TUDdeSnqq4WEWs1tm/RzQy7EuiC+Wohp Mbw+dt3uO2TTQrJ0wdv/txxsmESkg5lUicx2zkl8bqZosEsOmvbYjR0vnPnclQj/ v6514e4VL38nobJqniC92TvjEioWztzx9vckp72d90BEiqIJQlSdixOmP41IXlyX f/2vhWmoBnJH/Rt2/j2Big9ONnvHF69Q4jkAU3PIMu3GSqBCoH8UNZVwYUqXnBM3 R8N8xTKkjjoh/1Dn0him7JjGWbJoqK/3xhav/cKogUVL3kA5Kwizz3GVBfalreKc UGC/2ctQ9G17+WPe3hRD+MR6JsCaW+HwoVL2vmDa1aQyBrWQK8seyBiHrUdFtguB yKYAWrN9ybj72wh6LppTiF0WpvoPcDWHCo53RhRcyYo/IKw0hT69FFQt+HXdkqWs Lh/MT2blx6mfJwt6mRlJZIBEAq03LssKfIWyqMw+oETlq0QIiLY= =tIcU -----END PGP SIGNATURE----- --=_afcc090eeb35f4cae43fa1bcd152af2b--