Re: This is going to break port building without poudriere!

Hi Alexander,

On Tue, Jan 30, 2024 at 11:31 AM Alexander Leidinger <
Alexander@leidinger.net> wrote:

> Am 2024-01-30 10:42, schrieb Luca Pizzamiglio:
>
> > Hi Alexander.
> >
> > Subpackage modularization of existing ports (i.e. qt6-tools) provides
> > benefits to package users/builders: smaller dependency footprint,
> > smaller usage on disk (useful for embedded systems), smaller jails.
> > There are no benefits nor regressions for port builders for this
> > specific use case.
>
> Nicely worded. I agree to that. At the same time it is a regression for
> port builders. Installing from ports is not on par with installing from
> pkg anymore in this case. What you describe means we can use subpackages
> only for leaf ports. Ports which are a dependency can not converted to
> subpackages (in the sense of "no slave port available for the
> subpackages").
>
> I disagree that this is regression, but I agree that it's an unexpected
change.
Installing from ports can install more than installing from pkg.
I don't call it a regression, as it doesn't introduce bugs, ports can be
successfully built as before.
However, it introduces a divergence that requires more attention.

> On the other hand, moving master/slave ports to subpackages comes with
> > the cost of losing the ability to build the slave ports independently.
> > For package users there is no change and some benefit for package
> > builders.
> > The issue can be mitigated by introducing options to select which
> > subpackage to build (available for make install as well), but, still, a
> > single subpackage cannot be built independently.
>
> This is not a proper mitigation. I used mail/roundcube and lang/php as a
> specific example for this particular case where it is not a proper
> mitigation.
>
It's a mitigation, as you can reduce what you need to build, but it's not a
solution and it doesn't fix the breaking change.
It only offers a way to install less, but no way to install a subpackage
only.


> > This limitation _can_ be unacceptable in some cases. For example,
> > bofh@, the php maintainer, considers subpackages not the way to go, so
> > the master/slave approach will stay.
>
> Do you see that you just told that one of the best role models for
> subpackages according to you previous mail will not use subpackages
> because of the limitations I highlighted?
>
Yes, your arguments contributed to change my opinion on this point.
Is this a problem?


> > We introduced subpackages in the framework to explore all use cases, by
> > trying and testing its adoption.
> > By doing so we have already identified some issues (i.e. USES is not
> > subpackage-aware yet) and interesting new use cases (subpackage with
> > debug symbols).
>
> You are surely aware that you haven't mentioned one of the points I
> talked about as an issue, don't you? I have not seen any technical
> answer which shows that my technical description of issues is wrong. I
> want to make you aware that I understand the responses from you and the
> others in this thread as: "we do not care about those regressions and do
> not want to fix them" (I understand it like that because I see no
> acknowledgement of those issues, just answers which look like a
> smokescreen and pointing into directions of good faith). If you would
> acknowledge the technical issues highlighted in this thread (or show
> evidence that those regressions can not happen) and tell that the
> adoption of subpackages is monitored to not introduce those regressions
> I talk about, my understanding of the situation would change.
>

So you are saying that my answers are "smokescreen and pointing into
directions of good faith".
Am I trying to deceive you? Are you assuming that I have an ill intent that
needs to be masked?
Those seem serious accusations and they trouble me.

As per: "we don't care about those regressions and do not want to fix them".
I've already acknowledged that subpackages are going to introduce a
breaking change.
It's possible to have different outcomes if you use packages and ports.

We are going to have a portmgr meeting soon and we will see how to proceed.
The solutions you propose (subpackge+slave port) don't make any sense to
me, I would prefer to focus on a long term solution.
Technically, it should be possible to install a portion of a port.
It's more difficult to build only a part of a port, but this aspect is less
relevant.

> As per master/slave merge use case, we will let the maintainers decide
> > if they want to move forward with subpackage adoptions, knowing the
> > regression for port builders, but we won't push them in this direction.
>
> I consider it unfortunate that you describe it like that. I was hoping
> you would tell that portmgr will prevent the removal of slave packages
> and enforce the rule of having slave ports for subpackages aware ports
> to prevent regressions for users which install from ports (until the
> technical valid issues I have pointed out are resolved -- and they can
> be fixed, a first step would be to make the names of subpackages match
> normal packages names = replacing the '~' in the name with a '-' ASAP to
> prevent churn later).
>
> Note, in src some big discussion like this of having several committers
> identify regressions and no immediate fix would lead to a backout until
> it is fixed. I do not ask for a backout of this, but I ask for a strong
> lock/policy by portmgr on the subpackages feature like I already
> described in my previous mails (until the regressions the use of
> subpackages would create are fixed). This would allow for more
> experimentation by a lot of people without the need to patch the ports
> tree and without introducing regressions for a simple "make install" of
> ports with a lot of dependencies.
>

Are you asking to completely block the introduction of subpackages?
A complete block seems excessive to me, but some additional restriction can
be considered.

In any case, an update on the subpackage topic needs to be shared in the
next few days.

Best regards,
Luca

>
> Bye,
> Alexander.
>
> > Best regards,
> > Luca
> >
> > On Mon, Jan 29, 2024 at 10:04 AM Alexander Leidinger
> > <Alexander@leidinger.net> wrote:
> >
> >> Am 2024-01-27 16:59, schrieb Luca Pizzamiglio:
> >>
> >>> Hi Stefan.
> >>>
> >>> Let's start from the beginning, as it seems that things are not
> >>> clear.
> >>>
> >>> Subpackages is a feature to create multiple packages from one build
> >>> Those subpackages can depend on the main package.
> >>> The main package cannot depend on any subpackages.
> >>> This limits the cases where subpackages can be applied. The main
> >>> package MUST be independent from its subpackages. Subpackages can
> >>> only
> >>> add features (like a plugin).
> >>> To recall your NLS example (ref
> >>> https://reviews.freebsd.org/D16457#715443): this is not a use-case
> >>> for
> >>> subpackages. If the main program/library needs to be compiled
> >>> differently with or without NLS, this is not viable for subpackages.
> >>> If a port needs to be built multiple times to create different
> >>> subpackages, this is not a viable case for subpackages.
> >>> A good candidate is qt6-tools: this port contains multiple tools
> >>> (designer, linguist, help, and so on). Those tools could be put in
> >>> different subpackages.
> >>>
> >>> I hope this explanation helps to clarify and address some of your
> >>> concerns.
> >>
> >> As I read this, lang/php is the best example of a port where
> >> subpackages
> >> has a benefit (in the sense it matches your description above to the
> >> letter, the main port independent from the subpacakges, and what can
> >> be
> >> build as subpackages is a plugin/extension).
> >>
> >>> OPTIONS and SUBPACKAGES
> >>> Do we have to convert OPTIONS to SUBPACKAGES? No.
> >>> Can a SUBPACKAGE be built only if an OPTION is enabled? Yes.
> >>> The only viable use cases for SUBPACKAGES being enabled/disabled by
> >>> OPTIONS is limited to those portions of the port that do not affect
> >>> the
> >>> main package.
> >>> Examples are: additional libraries, additional data files, and so on.
> >>>
> >>> Consolidate master/slave ports in one bigger ports
> >>> About this topic, I guess your concerns are mainly related to
> >>> potential
> >>> explosion of build time of the consolidated ports.
> >>> This is a justified concern.
> >>> In those cases, we are suggesting to convert slave ports in
> >>> SUBPACKAGES
> >>> enabled via OPTIONS.
> >>> This will allow port builders to configure the bigger ports to not
> >>> build all SUBPACKAGES, but only the needed ones. This should restore
> >>> the previous build time.
> >>>
> >>> However, as for the php case, the maintainer is going to evaluate if
> >>> the consolidation makes sense or not.
> >>> If a consolidation is going to result more problematic than
> >>> beneficial,
> >>> it can be reverted and subpackages not adopted for the use case.
> >>
> >> If I understand the sum of all the above correct, you suggest to
> >> remove
> >> slave ports and to use subpackages instead (where this makes sense in
> >> terms of the current implementation of subpackages). Or worded
> >> differently to the same effect (as I only care about the effect and
> >> not
> >> the intention), when someone converts a port to subpackages, the
> >> corresponding slave packages shall be removed (or for new ports: slave
> >> ports shall not be introduced in this case).
> >>
> >> Removing slave ports means we can not depend upon specific parts
> >> anymore
> >> when installing from ports, as the subpackages can not be targeted for
> >> install directly and my example of a subpackages aware php results in
> >> security implications of to much being installed and active if
> >> installed
> >> via make install in /usr/ports/something/with_webinterface. -> best
> >> example of lang/php to use subpacakges is the best example of why not
> >> to
> >> use subpackages / shows what is a regression in the features we
> >> provide
> >> in our ports collection.
> >>
> >> While qt6-tools may be a good example where subpackages makes sense,
> >> we
> >> can not depend on subpacakges for "make install", and as if the port
> >> would be converted to have subpackages but no slave ports are
> >> introduced, pkg install and make install would differ in the amount of
> >> installed files.
> >>
> >>> For port builders
> >>>
> >>> Port builders can experience longer build times in the future, as
> >>> master/slave ports could be consolidated in one single larger ports.
> >>> If this is the case, the larger ports should provide OPTIONS to not
> >>> build unneeded subpackages.
> >>> If no OPTION is available, please work with the maintainer to
> >>> introduce
> >>> one.
> >>
> >> I fail to see the benefit:
> >> We either lose the possibility to target parts of a package (when
> >> slave
> >> ports are removed / not introduced) on make install (with a similar
> >> amount of build time for the ports tree as it is right now), or get
> >> higher build times for the package builders. In both cases we do not
> >> gain something significant.
> >>
> >> If we want to keep the (useful in some cases) feature to install from
> >> ports (there is the case of py39-rpds-py failing to build in my
> >> poudriere which I tried to debug with the author of py-maturin due to
> >> a
> >> runtime issue in maturin, which shows up in my the cross build on
> >> amd64-intel for amd64-athlon64... which in the end leads me to build
> >> py-rpds-py on the target machine from ports), the current
> >> implementation
> >> of subpackages has to come with the requirement to create slave ports
> >> for each subpackage.
> >>
> >> That's my concern, and that's the reason why I have the opinion, that
> >> portmgr has to keep the lock on subpackages and reject any subpackage
> >> which don't come with a slave port.
> >>
> >> I would be OK to lift this restriction, when the current
> >> implementation
> >> is changed to be able to only install the files of a subpacakge on
> >> make
> >> install (an implementation could be to require "make install
> >> TARGET_SUBPACAKGES=sub1,sub2,sub3" or "make install-subpackage1
> >> install-subpacakge2"... as long as recursive dependencies are handled
> >> according to this requirement, those people designing, implementing
> >> and
> >> reviewing this can argue about such details).
> >>
> >> Keeping the current implementation (with the restriction of always
> >> introducing slave packages for subpackages) would need a way to denote
> >> that a slave port covers a specific subpackage which would allow
> >> package
> >> builders to skip those slave ports (and the subpacakges would need to
> >> have the same package name as the slave port, no idea if this has a
> >> technical disadvantage in terms of having 2 different origins for the
> >> same package name, but it surely would be confusing for people at
> >> first
> >> look).
> >>
> >> TLDR: for the use cases you specified in the beginning, I do not see a
> >> benefit, only drawbacks. Can you please provide an example of a
> >> benefit
> >> I fail to see (yes, more modularity for qt6-tools may be beneficial
> >> for
> >> some people, but the cost/benefit between qt6-tools (something which
> >> we
> >> don't provide right now) and "make install" (what we provide right
> >> now)
> >> or "build time reduction for package builders" (which would have a
> >> benefit for a lot of use cases) is in my books much more in the
> >> direction of "make install" and "build time reduction" than towards
> >> the
> >> modularization of qt6-tools)?
> >>
> >> Bye,
> >> Alexander.
> >>
> >> --
> >> http://www.Leidinger.net Alexander@Leidinger.net: PGP
> >> 0x8F31830F9F2772BF
> >> http://www.FreeBSD.org    netchild@FreeBSD.org  : PGP
> >> 0x8F31830F9F2772BF
>
>
> --
> http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF
> http://www.FreeBSD.org    netchild@FreeBSD.org  : PGP 0x8F31830F9F2772BF
>