From nobody Thu Jan 25 19:57:49 2024 X-Original-To: ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TLWpQ3c7Gz57w2W for ; Thu, 25 Jan 2024 19:58:06 +0000 (UTC) (envelope-from luca.pizzamiglio@gmail.com) Received: from mail-io1-f41.google.com (mail-io1-f41.google.com [209.85.166.41]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TLWpQ1rB5z4Mg8; Thu, 25 Jan 2024 19:58:06 +0000 (UTC) (envelope-from luca.pizzamiglio@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-io1-f41.google.com with SMTP id ca18e2360f4ac-7bed9f0ea20so302372339f.2; Thu, 25 Jan 2024 11:58:06 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706212682; x=1706817482; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=H1ONWn1w95EWzaMW4OiH0zyTEDB4I54dugbT4iT/SwI=; b=W4HKReofLVKfhcGgszt+3DcrJA1cSGjONcwddKQtih0Zt4vEh5fquc+JRnrBXOgpOk bNlL+szedETpm5ygVRF5SZqE5q89M4mkrCUKAzmzEeLoIOva9cbaq004t5qZTfUjObCP Mm+rBWPCDJ5uwD7PHjMs6ggcWr6mvrn1aRqXqfzbyfAbdgE8IHxyVhzDWxTHQUd9C+aN DxESY2RoTtNRr8yXuQAz/I58neH4qlkoId+w74nbW9AEf14YuEXXg9j+1t+emziQoAr5 +4nLjffMAwbvifejdhYsWJXj6wQmGhrOPaAWWC9zjhX3/XSU5NlqqQbQoRp20aPWp7Kz eo9Q== X-Gm-Message-State: AOJu0YxGLxGL+Kh7pTURJhve5rQCVeCvxMclhBQ265MQ+SLPkvBbE4K0 zFuuiYhXTZ4Eu49CNDKrJc6L3k4YNr2dqCZ/OvYs9QkIQDu6uO/2/I0e3lZW X-Google-Smtp-Source: AGHT+IFatLZf7pyzlUEFTiknn6we7k89ntPOxG04fsE8KfKzFindapvnevK27tGoUv5JjzbCy3LfZg== X-Received: by 2002:a5d:9708:0:b0:7bf:5631:9283 with SMTP id h8-20020a5d9708000000b007bf56319283mr307823iol.29.1706212682271; Thu, 25 Jan 2024 11:58:02 -0800 (PST) Received: from mail-io1-f43.google.com (mail-io1-f43.google.com. [209.85.166.43]) by smtp.gmail.com with ESMTPSA id v22-20020a05663812d600b0046ed5f51becsm678458jas.39.2024.01.25.11.58.01 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 25 Jan 2024 11:58:01 -0800 (PST) Received: by mail-io1-f43.google.com with SMTP id ca18e2360f4ac-7bf98500c2cso176309339f.1; Thu, 25 Jan 2024 11:58:01 -0800 (PST) X-Received: by 2002:a05:6e02:526:b0:361:9612:3ce9 with SMTP id h6-20020a056e02052600b0036196123ce9mr259533ils.45.1706212681708; Thu, 25 Jan 2024 11:58:01 -0800 (PST) List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Luca Pizzamiglio Date: Thu, 25 Jan 2024 20:57:49 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: This is going to break port building without poudriere! (was: Subpackage explanations) To: Stefan Esser Cc: freebsd-ports , portmgr , FreeBSD Core Team Content-Type: multipart/alternative; boundary="000000000000cdfc6e060fca984e" X-Rspamd-Queue-Id: 4TLWpQ1rB5z4Mg8 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] --000000000000cdfc6e060fca984e Content-Type: text/plain; charset="UTF-8" Hi Stefan, I did reply to your first email, but not to your second one. I preferred (in agreement with portmgr@) to open the discussion with everyone, instead of keeping it just between you and me. As you can read in the email you have linked, I didn't ignore your comments. This implementation will break port dependencies, since there is no way > a port can depend on a specific sub-package - there even is no way a > non-default sub-package can be built without manual selection of the > options that activate its creation. > A port can depend on a specific sub-package. The category/origin~subpkg is the chosen format. Non-default subpackages shouldn't exist. In the aforementioned email, I explicitly say that IF a subpackage is enabled by an option, the option must be enabled by default. Your previous comments contributed to making this point clearer. > Dependencies stated in the port Makefile are converted into package > dependencies that can be resolved by the "pkg" command, but that cannot > be directly used to build and install the requested sub-package from > a port. > I don't know where you get this idea, but it's not how it works. If a port has a subpackage dependency, the category/origin~subpkg is the chosen format to express that dependency. The related port (category/origin) has to be built and installed. By running `make install`, the whole port is installed, subpackages as well. The only "issue" I see is that via 'make install' you cannot install only the subpackage, but the entire port only. However this is not different than before. Has there been a general consensus that support for direct port building > (without poudriere) will be abandoned? > Port building is not abandoned, and it's actively supported, as it's the foundation of poudriere. If a regression has been introduced, by me or anyone else, it has to be fixed. I may have overlooked some use cases, but AFAIK I didn't introduce any regression (confirmed by the exp-run) > Ports that do not create sub-packages can still be depended on by other > ports, but as critical dependencies have been depended to sub-packages > a ever large fraction of ports will only build in poudriere. > Again, no. If you run make install on any port, it will be built. The package and the subpackages are all installed. You cannot install a portion of a port, and it has never been possible. Nothing is broken here and I fail to understand where you get the idea of this behavior. I wrote tests and examples to implement the feature and get them working before adding support to poudriere. This change does also obviously break port management tools like portmaster, > which took me significant effort to adapt to FLAVOR support (which also had > been implemented without consideration for other tools than poudriere), and > which I have been maintaining since then. > Yes, portmaster need to be able to parse the new dependencies format, by removing the ~subpkg. By removing the suffix, you get the category/port and everything is fine. As announced, we are keeping subpackages adoption blocked with a git-hook to give time to maintainers to add support to subpkg and to introduce the feature slowly. portmgr@ doesn't officially support neither portmaster nor portupgrade. We simply lack the manpower to do it. The reason given for sub-packages support is reduced build time for some > packages that share a common distfile (e.g. qt5) when building official > packages. > As explained, in the short term there are this benefit and getting rid of many master/slave ports (hard to maintain) But there are several use cases in the future. For instance, work is in place to provide debug symbols as subpackage, in a similar way as pkg-base. > But this comes at a high cost for all builds outside the package build > cluster, since now lots of unnecessary sub-trees will be compiled and > installed, if only one program (i.e. sub-package) is desired. > This is the reason why OPTIONS with SUBPACAKGES have been introduced, to reduce build time for port builders. > You are pessimizing the build for thousands of users to spare a few > cycles for a very small percentage of packages built once in a while on > the build cluster. > I have already explained this point above. I had pointed out other issues with this approach in the review comment > and in the private mail I sent 2024-01-02 after finding that this version > has been committed without formal acceptance of your review D40549. > As I replied to you already, there has been no formal acceptance in phabricator, but there was consensus in portmgr@ to land it. I apologize for not having used the appropriate reviews channel, I totally agree that it has not been a good behavior from my side, as I'm not providing a good example. You are breaking use-cases of a large number of users that still build > ports without poudriere. This is especially causing a barrier to entry > of new port developers, since this will require them to setup poudriere > before they can begin port development. (This will only become an issue > over time, as more and more dependencies will have been converted to > sub-packages, but then there will be no way back.) > I still fail to see what is going to break. Non-poudriere users can experience longer build time for dependencies, but dependencies can be installed via packages, at least most of the time. Your announcement mentions some of the issues, but does not offer any > actual solution. > A sane implementation of sub-packages would not make their creation > depend on OPTION settings, with no way to determine the required OPTION > setting for a non-default sub-package (i.e., only default sub-packages > can be depended on). > > I'm not going to repeat all the issues pointed out in > > https://reviews.freebsd.org/D16457#715443 > > but had on multiple occasions pointed out that a sane mechanism would > start with "SUB_PACKAGE_DEFINE" and "SUB_PACKAGE_DEFAULT" variables and > with a mechanism to determine the required OPTION values from the actual > set of sub-packages to be built (instead of the opposite, to make the > selection of sub-packages depend on the OPTIONs). > I clearly addressed this topic in the aforementioned email. We chose not to follow this approach, and I motivated it (changing configuration of other ports it's NOT something to have IMO). We clearly disagree on this point. I have opened up the discussion to get feedback from the rest of the community as well. So far, you are the only one strongly against this approach. Best regards, pizzamig --000000000000cdfc6e060fca984e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Stefan,

= I did reply to your first email, but not to your second one.
I preferre= d (in agreement with portmgr@) to open the discussion with everyone, instea= d of keeping it just between you and me.

As yo= u can read in the email you have linked, I didn't ignore your comments.=

This implementation will break port dependencies, since there is no way
a port can depend on a specific sub-package - there even is no way a
non-default sub-package can be built without manual selection of the
options that activate its creation.
A port can depend = on a specific sub-package. The category/origin~subpkg is the chosen format.=
Non-default subpackages shouldn't exist. In the aforemention= ed email, I explicitly say that IF a subpackage is enabled by an option, th= e option must be enabled by default.
Your previous comments contributed = to making this point clearer.
=C2=A0
Dependencies stated in the port Makefile are converted into package
dependencies that can be resolved by the "pkg" command, but that = cannot
be directly used to build and install the requested sub-package from
a port.
I don't know where you get this idea, but = it's not how it works.
If a port has a subpackage d= ependency, the category/origin~subpkg is the chosen format to express that = dependency.
The related port (category/origin) has to be built and inst= alled.
By running `make install`, the whole port is installed, su= bpackages as well.
The only "issue" I see is that = via 'make install' you cannot install only the subpackage, but the = entire port only.
However this is not different than before.<= br>

Has there been a general consensus that support for direct port building (without poudriere) will be abandoned?
Port building i= s not abandoned, and it's actively supported, as it's the foundatio= n of poudriere.
If a regression has been introduced, by me or any= one else, it has to be fixed.
I may have overlooked some use case= s, but AFAIK I didn't introduce any regression (confirmed by the exp-ru= n)
=C2=A0
Ports that do not create sub-packages can still be depended on by other
ports, but as critical dependencies have been depended to sub-packages
a ever large fraction of ports will only build in poudriere.
Again, no.
If you run make install on any port, it will b= e built. The package and the subpackages are all installed.
You c= annot install a portion of a port, and it has never been possible.
Nothing is broken here and I fail to understand where you get the idea of= this behavior.
I wrote tests and examples to implement the featu= re and get them working before adding support to poudriere.
<= br>
This change does also obviously break port management tools like portmaster= ,
which took me significant effort to adapt to FLAVOR support (which also had=
been implemented without consideration for other tools than poudriere), and=
which I have been maintaining since then.
Yes, portmas= ter need to be able to parse the new dependencies format, by removing the ~= subpkg. By removing the suffix, you get the category/port and everything is= fine.
As announced, we are keeping subpackages adoption bloc= ked with a git-hook to give time to maintainers to add support to subpkg an= d to introduce the feature slowly.
portmgr@ doesn't offic= ially support neither portmaster nor portupgrade. We simply lack the manpow= er to do it.

The reason given for sub-packages support is reduced build time for some packages that share a common distfile (e.g. qt5) when building official
packages.
As explained, in the short term there are th= is benefit and getting rid of many master/slave ports (hard to maintain)
But there are several use cases in the future.
For in= stance, work is in place to provide debug symbols as subpackage, in a simil= ar way as pkg-base.
=C2=A0
But this comes at a high cost for all builds outside the package build
cluster, since now lots of unnecessary sub-trees will be compiled and
installed, if only one program (i.e. sub-package) is desired.
This is the reason why OPTIONS with SUBPACAKGES have been introduce= d, to reduce build time for port builders.
=C2=A0
You are pessimizing the build for thousands of users to spare a few
cycles for a very small percentage of packages built once in a while on
the build cluster.
I have already explained this point= above.

I had pointed out other issues with this approach in the review comment
and in the private mail I sent 2024-01-02 after finding that this version has been committed without formal acceptance of your review D40549.
As I replied to you already, there has been no formal accepta= nce in phabricator, but there was consensus in portmgr@ to land it.
I apologize for not having used the appropriate reviews channel, I total= ly agree that it has not been a good behavior from my side, as I'm not = providing a good example.

You are breaking use-cases of a large number of users that still build
ports without poudriere. This is especially causing a barrier to entry
of new port developers, since this will require them to setup poudriere
before they can begin port development. (This will only become an issue
over time, as more and more dependencies will have been converted to
sub-packages, but then there will be no way back.)
I s= till fail to see what is going to break.
Non-poudriere users can exper= ience longer build time for dependencies, but dependencies can be installed= via packages, at least most of the time.
<= br>
Your announcement mentions some of the issues, but does not offer any
actual solution.

A sane implementation of sub-packages would not make their creation
depend on OPTION settings, with no way to determine the required OPTION
setting for a non-default sub-package (i.e., only default sub-packages
can be depended on).

I'm not going to repeat all the issues pointed out in

=C2=A0 =C2=A0 =C2=A0 =C2=A0 https://reviews.freebsd.org/D16= 457#715443

but had on multiple occasions pointed out that a sane mechanism would
start with "SUB_PACKAGE_DEFINE" and "SUB_PACKAGE_DEFAULT&quo= t; variables and
with a mechanism to determine the required OPTION values from the actual set of sub-packages to be built (instead of the opposite, to make the
selection of sub-packages depend on the OPTIONs).
I cl= early addressed this topic in the aforementioned email.
We chose = not to follow this approach, and I motivated it (changing configuration of = other ports it's NOT something to have IMO).
We clearly d= isagree on this point.
I have opened up the discussion to get fee= dback from the rest of the community as well.
So far, you are= the only one strongly against this approach.

Best= regards,
pizzamig

--000000000000cdfc6e060fca984e--