Re: Problem with the package builds

On Jul 10, 2023, at 21:27, Rainer Hurling <rhurlin@gwdg.de> wrote:

> Am 10.07.23 um 23:26 schrieb Kevin Oberman:
>> On Mon, Jul 10, 2023 at 10:31 AM Rainer Hurling <rhurlin@gwdg.de <mailto:rhurlin@gwdg.de>> wrote:
>>    Am 10.07.23 um 08:15 schrieb Rainer Hurling:
>>     > Am 10.07.23 um 08:08 schrieb Mark Millard:
>>     >> On Jul 9, 2023, at 22:52, Rainer Hurling <rhurlin@gwdg.de
>>    <mailto:rhurlin@gwdg.de>> wrote:
>>     >>
>>     >>> Am 10.07.23 um 06:53 schrieb Mark Millard:
>>     >>>> Konstantin Belousov <kostikbel_at_gmail.com
>>    <http://kostikbel_at_gmail.com>> wrote on
>>     >>>> Date: Mon, 10 Jul 2023 01:18:05 UTC :
>>     >>>>> On Sun, Jul 09, 2023 at 05:52:50PM -0700, Kevin Oberman wrote:
>>     >>>>>> Something looks really messed up on the beefy16 package
>>    builder for
>>     >>>>>> 13.1amd64.
>>     >>>>>>
>>     >>>>>> It is building about 9500 packages and started almost two days
>>     >>>>>> ago. It has
>>     >>>>>> only completed 459 packages, a rate of 10 packages/hour and,
>>    if it
>>     >>>>>> continues at this rate, will not complete for about 40 days. Nt
>>     >>>>>> sure if
>>     >>>>>> this was triggered by the recent updated to pkg, something that
>>     >>>>>> may have
>>     >>>>>> changed in poudriere or in the configuration, but almost all
>>    jails
>>     >>>>>> are are
>>     >>>>>> in one of the depends states, lib-depends, build-depends,
>>     >>>>>> run-depends,
>>     >>>>>> often running for hours.
>>     >>>>>> E.g. gnome-desktop-42.4_1 is in lib_depends and has been
>>    building for
>>     >>>>>> 1:09:59. Last build of the port took about 55 minutes. ATM
>>    there
>>     >>>>>> are 7
>>     >>>>>> lib-depends builds, 5 build-depends, and 1 run-depends
>>    building.
>>     >>>>>> No jail is
>>     >>>>>> actually boulding the package, itself.
>>     >>>>>>
>>     >>>>>> Wild guess... It is rebuilding all depends every time. In any
>>     >>>>>> case, someone
>>     >>>>>> needs to look at this.
>>     >>>>>>
>>     >>>>>> Build report is at
>>     >>>>>>
>>    http://beefy16.nyi.freebsd.org/build.html?mastername=131amd64-default&build=e81dafc952d7 <http://beefy16.nyi.freebsd.org/build.html?mastername=131amd64-default&build=e81dafc952d7>
>>     >>>>>
>>     >>>>> I noted that 'pkg-static install' seems to be much slower
>>    locally.
>>     >>>>> Not looked into it.
>>     >>>> Summarizing the later observations:
>>     >>>> Slow: pre-1400000 Jail OSVERSION and recent ports (such as pkg)
>>     >>>> ("default")
>>     >>>> Fast: pre-1400000 Jail OSVERSION and older ports  (such as pkg)
>>     >>>> ("quarterly")
>>     >>>> Fast: 1400093     Jail OSVERSION and recent ports (such as pkg)
>>     >>>> ("default")
>>     >>>> (Host OSVERSION 1400089 is a common context across them all.)
>>     >>>> Details:
>>     >>>> The list of build contexts with 50+ hr Elapsed but under 500
>>     >>>> ports built looks like:
>>     >>>> All started on Sat, 08 Jul 2023 01:0[12]:?? GMT:
>>     >>>> beefy6 124amd64 default
>>     >>>> Host OSVERSION: 1400089
>>     >>>> Jail OSVERSION: 1204000
>>     >>>> beefy15 131i386 default
>>     >>>> Host OSVERSION: 1400089
>>     >>>> Jail OSVERSION: 1301000
>>     >>>> beefy16 131amd64 default
>>     >>>> Host OSVERSION: 1400089
>>     >>>> Jail OSVERSION: 1301000
>>     >>>> But the following 2 built over 5700 ports in under 50 hr Elapsed.
>>     >>>> Both started on Sat, 08 Jul 2023 01:01:?? GMT:
>>     >>>> beefy14 131amd64 quarterly
>>     >>>> Host OSVERSION: 1400089
>>     >>>> Jail OSVERSION: 1301000
>>     >>>> beefy13 131i386 quarterly
>>     >>>> Jail OSVERSION: 1301000
>>     >>>> Ignoring armv6's qemu based builds, the more recently started
>>     >>>> main-amd64 and main-arm64 builds do not show the problem.
>>     >>>> Started Sat, 08 Jul 2023 01:29:09 GMT and built over 4400 ports
>>     >>>> in under 51 hr:
>>     >>>> beefy18 main-amd64 default
>>     >>>> Host OSVERSION: 1400089
>>     >>>> Jail OSVERSION: 1400093
>>     >>>> Started Sun, 09 Jul 2023 03:11:16 GMT 25:00:07 GMT and built over
>>     >>>> 1500 ports in under 26 hrs:
>>     >>>> ampere2 main-arm64 default
>>     >>>> Host OSVERSION: 1400089
>>     >>>> Jail OSVERSION: 1400093
>>     >>>> ===
>>     >>>> Mark Millard
>>     >>>> marklmi at yahoo.com <http://yahoo.com>
>>     >>>
>>     >>> Since this does not seem to be mentioned in the previous posts.
>>    The
>>     >>> phenomenon does not only occur in the pkg clusters.
>>     >>>
>>     >>> In fact, I also observe this severely delayed behavior under
>>     >>> Poudriere on my very current 14.0-CURRENT box. All *-depends
>>    sections
>>     >>> need a multiple of the usual times ...
>>     >>>
>>     >>
>>     >> So, something like:
>>     >>
>>     >> Host OSVERSION: 1400093
>>     >> Jail OSVERSION: 1400093
>>     >> and:
>>     >> /packages/All/pkg-1.20.1_1.pkg
>>     >
>>     > Yes, this corresponds exactly to my versions of the system, the
>>    jail and
>>     > ports-mgmt/pkg.
>>    The newest update of ports-mgmt/pkg to version 1.20.2 by bapt@ solves
>>    the problem for me :)
>> Great! So, can anyone restart the build or kill it until it will start again at 1:00 UTC tomorrow morning or will it start building with the updated pkg? I see no signs of using any change to this point.
> 
> As I understand it, the ports-mgmt/pkg of the system running Poudriere must be updated beforehand?
> 
> At least on my side, this seems to work as expected :)
> 

poudriere builds pkg updates first (if needed) and then uses the pkg it
built for building the later ports into packages.

But, after the restarts of main-* builds, the FreeBSD build servers are
still showing examples were, after an 1hr, some builds are still in
build-depends. Also there was an example I saw were after 1.5 hr it was
still in run-depends.

It may be that things are improved but not fully fixed relative to
some performance issues.

===
Mark Millard
marklmi at yahoo.com