Re: This is going to break port building without poudriere!
Date: Fri, 26 Jan 2024 16:08:04 UTC
Am 25.01.24 um 20:57 schrieb Luca Pizzamiglio:
> Hi Stefan,
>
> I did reply to your first email, but not to your second one.
> I preferred (in agreement with portmgr@) to open the discussion with everyone,
> instead of keeping it just between you and me.
>
> As you can read in the email you have linked, I didn't ignore your comments.
You have mentioned some of the issues without providing a solution.
> This implementation will break port dependencies, since there is no way
> a port can depend on a specific sub-package - there even is no way a
> non-default sub-package can be built without manual selection of the
> options that activate its creation.
>
> A port can depend on a specific sub-package. The category/origin~subpkg is the
> chosen format.
> Non-default subpackages shouldn't exist. In the aforementioned email, I
> explicitly say that IF a subpackage is enabled by an option, the option must be
> enabled by default.
> Your previous comments contributed to making this point clearer.
If there should be no "non-default subpackages", then selecting subpackages
to install by OPTIONs makes no sense. But there are many scenarios where
not all subpackages can be built at the same time due to conlflicting
dependencies or requirements (often controlled by OPTIONs).
> Dependencies stated in the port Makefile are converted into package
> dependencies that can be resolved by the "pkg" command, but that cannot
> be directly used to build and install the requested sub-package from
> a port.
>
> I don't know where you get this idea, but it's not how it works.
> If a port has a subpackage dependency, the category/origin~subpkg is the chosen
> format to express that dependency.
Yes, but how will this dependency be processed in the ports system?
Will the port at the given origin be built (all subpackages) and all the
available subpackages will be installed?
How do you deal with the case, where o ne of those subpackages conflicts
with an already installed port?
What actually works is the mapping of the origin~subpackage into a package
dependency that can be resolved by the pkg command. That's what I am
complaining about: only sucgh package dependencies continue to work, but
port dependencies don't in the same way as without subpackages.
> The related port (category/origin) has to be built and installed.
> By running `make install`, the whole port is installed, subpackages as well.
> The only "issue" I see is that via 'make install' you cannot install only the
> subpackage, but the entire port only.
> However this is not different than before.
Except that before there was not one port that built potentially tens of
subpackages.
> Has there been a general consensus that support for direct port building
> (without poudriere) will be abandoned?
>
> Port building is not abandoned, and it's actively supported, as it's the
> foundation of poudriere.
The case of poudriere is not generic port building, since all dependencies
are installed from packages in poudriere. Poudriere never installs from a
port, only from packages, and it can deal with subpackage dependencies.
Port building on the base system uses a completely different approach than
package building in poudriere.
> If a regression has been introduced, by me or anyone else, it has to be fixed.
> I may have overlooked some use cases, but AFAIK I didn't introduce any
> regression (confirmed by the exp-run)
The exp-run is performed by building packages with poudriere. What has this
to do with the port building on the base system that I'm worried about?
Besides, you only showed that you did not break the ports system without
subpackages by adding features for the support of subpackages to bsd.port.mk.
The relevant test case would have been an exp-run with several relevant
ports converted to subpackages.
But even then, port building outside poudriere is different and may be
severely broken without any issues reported by an exp-run, even one that
tests actual subpackages.
> Ports that do not create sub-packages can still be depended on by other
> ports, but as critical dependencies have been depended to sub-packages
> a ever large fraction of ports will only build in poudriere.
>
> Again, no.
> If you run make install on any port, it will be built. The package and the
> subpackages are all installed.
Yes, and that is wrong!
> You cannot install a portion of a port, and it has never been possible.
No, but with subpackages it will become possible to install a portion of
a port from a package. And since the goal is to convert ports that share
a distfile to subpackages, this will become a common case for large ports.
> Nothing is broken here and I fail to understand where you get the idea of this
> behavior.
Because I consider just the above as breaking important features of the
current ports system (the ability to selectively install just the ports
that I need and not all that can potentially be built from one distfile).
> I wrote tests and examples to implement the feature and get them working before
> adding support to poudriere.
I have seen your tests and they are only relevant for the package dependency
case (dependencies resolved by the pkg command).
> This change does also obviously break port management tools like portmaster,
> which took me significant effort to adapt to FLAVOR support (which also had
> been implemented without consideration for other tools than poudriere), and
> which I have been maintaining since then.
>
> Yes, portmaster need to be able to parse the new dependencies format, by
> removing the ~subpkg. By removing the suffix, you get the category/port and
> everything is fine.
No, I get a whole lot of subpackages that I did not ask for!
> As announced, we are keeping subpackages adoption blocked with a git-hook to
> give time to maintainers to add support to subpkg and to introduce the feature
> slowly.
The damage will be to other users than the committers of these ports, and
even developers that depend on them might not notice any issues, since they
are using poudriere to test their ports.
> portmgr@ doesn't officially support neither portmaster nor portupgrade. We
> simply lack the manpower to do it.
I do not expect PortMgr to support either of these tools. But I do expect
PortMgr to not break features of the port system due to neglect of building
ports on the base system. I know that there have been discussions about
abandoning the base system builds (especially since in some cases ports do
build in a clean jail, but not with a previous version installed, e.g. because
of preferring installed header files above those in the source director<).
But most ports build just fine on the base system and this is the fastest way
for a developer to get some required functionality with free choice of the
OPTIONs (when different from those of a pre-built package, for example).
> The reason given for sub-packages support is reduced build time for some
> packages that share a common distfile (e.g. qt5) when building official
> packages.
>
> As explained, in the short term there are this benefit and getting rid of many
> master/slave ports (hard to maintain).
This does only work if there is e.g. a port that can be compiled for CLI and
GUI, for example, but that does not need special build options to include
the required hooks to the added functionality.
I have seen the devel/appstream port as of https://reviews.freebsd.org/D43445,
and it clearly shows the weakness of the subpackages implementation. You did
not get rid of a master/slave port combination, but you have now combined the
CLI and Qt6 versions (while there previously was one port for the CLI version
and one flavored port for Qt5/Qt6).
And devel/appstream/Makefile has become a lot more complex and the complexity
of having both a slave port and subpackages makes it even worse.
If I build the CLI version after subpackages, I get all of Qt6 as a dependency!
And Qt6 is not only fetched and built, but also installed (when building the
port as a dependency on the base system), whether I need it or not.
> But there are several use cases in the future.
> For instance, work is in place to provide debug symbols as subpackage, in a
> similar way as pkg-base.
There is nothing wrong with such use-cases, and I do not object the support
of subpackages at all.
I had just been hoping that my comments to the earlier review would lead to
a better design - which would have easily been possible (and I had worked
out a concept several years ago, which I had discussed with some members
of PortMgr and possibly also in mails to the ports maillist).
> But this comes at a high cost for all builds outside the package build
> cluster, since now lots of unnecessary sub-trees will be compiled and
> installed, if only one program (i.e. sub-package) is desired.
>
> This is the reason why OPTIONS with SUBPACAKGES have been introduced, to reduce
> build time for port builders.
But that does not work well if there is no way to deduce the required
OPTIONS from the required subpackages. And it requires manual intervention.
There should have been variables like e.g. SUBPKG_DEFINE; SUBPKG_DEFAULT,
SUBPKG_SELECT which control the building and installation of subpackages,
and OPTIONs might depend on the selected set of subpackages (e.g., if a
~docs subpackage is selected, the DOCS option could be acticated).
> You are pessimizing the build for thousands of users to spare a few
> cycles for a very small percentage of packages built once in a while on
> the build cluster.
>
> I have already explained this point above.
I did not see a valid explanation. Thousands (or more) port builders will
waste cycles and space for subpackages that are not actually wanted, since
there is no mechanism that allows to easily restrict the amount built
(without resorting to OPTIONs, but those are sticky and the next port to
be built may require subpackages controlled by other options, this is not
a workable conept).
> I had pointed out other issues with this approach in the review comment
> and in the private mail I sent 2024-01-02 after finding that this version
> has been committed without formal acceptance of your review D40549.
>
> As I replied to you already, there has been no formal acceptance in
> phabricator, but there was consensus in portmgr@ to land it.
> I apologize for not having used the appropriate reviews channel, I totally
> agree that it has not been a good behavior from my side, as I'm not providing a
> good example.
As explained before, I had mostly given up on FreeBSD in late 2022, mostly
because of frustration about PortMgr. I had stopped reading most of the
FreeBSD mail lists and might have missed an early announcement of the
plan to commit your Phabricator review. As you know, I had hoped to be
able to stear the subpackages development in a better direction as part
of the PortMgr team, but there was an objection of the author of the
original subpackages review (that your commit is based on), probably
because of my earlier comment on his approach.
> You are breaking use-cases of a large number of users that still build
> ports without poudriere. This is especially causing a barrier to entry
> of new port developers, since this will require them to setup poudriere
> before they can begin port development. (This will only become an issue
> over time, as more and more dependencies will have been converted to
> sub-packages, but then there will be no way back.)
>
> I still fail to see what is going to break.
> Non-poudriere users can experience longer build time for dependencies, but
> dependencies can be installed via packages, at least most of the time.
This "dependencies can be installed via packages" is just the sentiment of
PortMgr that has no interest for port building in the base system.
Some dependencies can be installed from packages, but in many cases that
leads to lots of already installed programs to be substituted with packages
built with different OPTIONs. Mixing packages and ports is not working well,
and the only sane possibilities are a local repository built with custom
options using poudriere, or just building everything from a port on the
base system.
> Your announcement mentions some of the issues, but does not offer any
> actual solution.
>
>
> A sane implementation of sub-packages would not make their creation
> depend on OPTION settings, with no way to determine the required OPTION
> setting for a non-default sub-package (i.e., only default sub-packages
> can be depended on).
>
> I'm not going to repeat all the issues pointed out in
>
> https://reviews.freebsd.org/D16457#715443
> <https://reviews.freebsd.org/D16457#715443>
>
> but had on multiple occasions pointed out that a sane mechanism would
> start with "SUB_PACKAGE_DEFINE" and "SUB_PACKAGE_DEFAULT" variables and
> with a mechanism to determine the required OPTION values from the actual
> set of sub-packages to be built (instead of the opposite, to make the
> selection of sub-packages depend on the OPTIONs).
>
> I clearly addressed this topic in the aforementioned email.
Sorry, I did not see these points addressed. Please give specific pointers.
IMHO, all of the issues in my comment to review D16457 are still open.
> We chose not to follow this approach, and I motivated it (changing
> configuration of other ports it's NOT something to have IMO).
What are you talking about ("changing configuration of other ports") ???
> We clearly disagree on this point.
> I have opened up the discussion to get feedback from the rest of the community
> as well.
> So far, you are the only one strongly against this approach.
Maybe this is because other developers will only notice the effects of this
change when many ports have been converted to subpackages?
This will make it much harder for new ports contributors to work on their
first port, since port builds outside poudriere will become penalized more
and more over time, as subpackages are introduced (and become dependencies).
I have deep insight into the ports framework since I needed to understand it
when I took over portmaster and added FLAVOR support to it. And as a certified
Common Criteria evaluator (plus other standards) I might have a different view
on aspects of software development.
I have mostly given up my attempts to help improve the ports system, for lack
of support (or at least not being ignored when I ask for a review) by PortMgr.
There are a lot of sometimes trivial changes, but my past experience with
PortMgr is that I cannot hope for any kind of review to suggested changes,
since everybody is busy working in his field of interest and does not care
for any outside contributions (and my attempt to contribute from within
PortMgr has been blocked, too).
Best regards, STefan