Re: We need to do something about build times

Hi,

Many interesting thoughts about current situation, here's my take on
it and I'm also trying to catch up.

Some ports do require quite a lot of CPU time and memory to build, I
fully understand that not everyone is using the latest hardware
available but there's little we can do about upstream projects overall
since the ports tree is after all intended for packaging not
maintaining. By that I mean there will likely be little to no action
taken if you report upstream that lets say Rust don't build with
12Gbyte of RAM. I do agree that if we can consolidate dependencies
it's for the better in terms of build times, disk footprint (for
installed ports, might not be much of an issue these days) and
potential overlapping libraries/dependencies. What we need to accept
is that 4 cores and 16Gb RAM isn't a "powerhouse" that will build
whatever you throw at it, that's the reality. We can't adapt ports to
some arbitrary "hardware requirement", that's not viable nor do we
have anywhere near the manpower to do so however if we can streamline
the build with little effort by all means, go for it.

What I have concerns about is adapting port options that by default
potentially will harm performance and disk footprint in favour of "it
builds faster". I would at least like to claim that we primarily use
FreeBSD to run applications not to build ports. If we can combine both
that would be great but the former should be prioritized over latter.
If we can make packages smaller in size we should because it benefits
"everyone" however if it is of a very little gain and adds a lot of
time to the build it should be evaluated. For example we do apply LTO
and a few other optimizations to all Rust (Cargo based) ports because
it greatly reduces binary size and in some cases also increases
peformance while increasing total build by a bit. There are already
other distros looking into enabling LTO "treewide" so we're likely
going to head there too eventually.

Porters Handbook covers unbundling and is also a policy for several
other distros. I think we do a pretty good about about it in general
however some projects makes it very hard or impossible to unbundle
heavy dependencies, all we can do here is to discuss it with upstream.
While we could "maintain" our own set of patches it's for certain a
recipe for ports getting abandoned due to maintence burden sooner than
later. In short, please upstream and have a discussion with upstream
about such concerns.

We already miss out in some cases quite significant performance
enhancements with pre-built packages for ports that do not support
run-time detection of hardware capatibilities so we probably need to
look at difference target branches for each arch futher down the road
adding more overall load.

Some observations on my end, 

Without carefully measuring I do find that my Poduriere instances in
many cases spends quite a bit of time not building but spending quite
a bit of time in configure stage and struggle to scale over ~4-6 jobs.

What we can do per port level that improves build times and scaling in
general: 

If upstream uses GNU Autotools, use upstream release archives as they
usually contains a configure script ready to run which means that you
can avoid USES= autoreconf which is slow and adds unncessary
dependencies.

If upstream provides an alternative to GNU Autotools and/or gmake,
libtool such as CMake or Meson it will likely run configure stage
faster and build quicker, sometimes by quite a bit especially if you
have a system with many "slow" cores (such as ARM but even x86 to some
degree).  On some ports you can cut down build times ~40-50% because
of better scaling and processing, avoiding USES= autoreconf and
friends. You likely wont see that much of a difference overall but
10-20% isn't impossible (slow systems such as RockPro64 tend to see
better improvements). There might be some quirks switching build
framework that you might need to work out with upstream but it's
usually worth the effort in the end. Not saying that it'll compensate
for spending 8h build LLVM/Clang but if we can "convert" lets say100
ports or more we will probably reduice the overall build time by quite
a bit. 

If dependencies are unbundled you can save I/O and processing time by
not extracting.

Example:
https://cgit.freebsd.org/ports/tree/net-mgmt/netdata/Makefile#n32

Best regards,

Daniel (diizzy@)