Re: Problem with the package builds

Kevin Oberman <rkoberman@gmail.com> writes:

> On Mon, Jul 10, 2023 at 10:31 AM Rainer Hurling <rhurlin@gwdg.de> wrote:
>
>> Am 10.07.23 um 08:15 schrieb Rainer Hurling:
>> > Am 10.07.23 um 08:08 schrieb Mark Millard:
>> >> On Jul 9, 2023, at 22:52, Rainer Hurling <rhurlin@gwdg.de> wrote:
>> >>
>> >>> Am 10.07.23 um 06:53 schrieb Mark Millard:
>> >>>> Konstantin Belousov <kostikbel_at_gmail.com> wrote on
>> >>>> Date: Mon, 10 Jul 2023 01:18:05 UTC :
>> >>>>> On Sun, Jul 09, 2023 at 05:52:50PM -0700, Kevin Oberman wrote:
>> >>>>>> Something looks really messed up on the beefy16 package builder for
>> >>>>>> 13.1amd64.
>> >>>>>>
>> >>>>>> It is building about 9500 packages and started almost two days
>> >>>>>> ago. It has
>> >>>>>> only completed 459 packages, a rate of 10 packages/hour and, if it
>> >>>>>> continues at this rate, will not complete for about 40 days. Nt
>> >>>>>> sure if
>> >>>>>> this was triggered by the recent updated to pkg, something that
>> >>>>>> may have
>> >>>>>> changed in poudriere or in the configuration, but almost all jails
>> >>>>>> are are
>> >>>>>> in one of the depends states, lib-depends, build-depends,
>> >>>>>> run-depends,
>> >>>>>> often running for hours.
>> >>>>>> E.g. gnome-desktop-42.4_1 is in lib_depends and has been building
>> for
>> >>>>>> 1:09:59. Last build of the port took about 55 minutes. ATM there
>> >>>>>> are 7
>> >>>>>> lib-depends builds, 5 build-depends, and 1 run-depends building.
>> >>>>>> No jail is
>> >>>>>> actually boulding the package, itself.
>> >>>>>>
>> >>>>>> Wild guess... It is rebuilding all depends every time. In any
>> >>>>>> case, someone
>> >>>>>> needs to look at this.
>> >>>>>>
>> >>>>>> Build report is at
>> >>>>>>
>> http://beefy16.nyi.freebsd.org/build.html?mastername=131amd64-default&build=e81dafc952d7
>> >>>>>
>> >>>>> I noted that 'pkg-static install' seems to be much slower locally.
>> >>>>> Not looked into it.
>> >>>> Summarizing the later observations:
>> >>>> Slow: pre-1400000 Jail OSVERSION and recent ports (such as pkg)
>> >>>> ("default")
>> >>>> Fast: pre-1400000 Jail OSVERSION and older ports  (such as pkg)
>> >>>> ("quarterly")
>> >>>> Fast: 1400093     Jail OSVERSION and recent ports (such as pkg)
>> >>>> ("default")
>> >>>> (Host OSVERSION 1400089 is a common context across them all.)
>> >>>> Details:
>> >>>> The list of build contexts with 50+ hr Elapsed but under 500
>> >>>> ports built looks like:
>> >>>> All started on Sat, 08 Jul 2023 01:0[12]:?? GMT:
>> >>>> beefy6 124amd64 default
>> >>>> Host OSVERSION: 1400089
>> >>>> Jail OSVERSION: 1204000
>> >>>> beefy15 131i386 default
>> >>>> Host OSVERSION: 1400089
>> >>>> Jail OSVERSION: 1301000
>> >>>> beefy16 131amd64 default
>> >>>> Host OSVERSION: 1400089
>> >>>> Jail OSVERSION: 1301000
>> >>>> But the following 2 built over 5700 ports in under 50 hr Elapsed.
>> >>>> Both started on Sat, 08 Jul 2023 01:01:?? GMT:
>> >>>> beefy14 131amd64 quarterly
>> >>>> Host OSVERSION: 1400089
>> >>>> Jail OSVERSION: 1301000
>> >>>> beefy13 131i386 quarterly
>> >>>> Jail OSVERSION: 1301000
>> >>>> Ignoring armv6's qemu based builds, the more recently started
>> >>>> main-amd64 and main-arm64 builds do not show the problem.
>> >>>> Started Sat, 08 Jul 2023 01:29:09 GMT and built over 4400 ports
>> >>>> in under 51 hr:
>> >>>> beefy18 main-amd64 default
>> >>>> Host OSVERSION: 1400089
>> >>>> Jail OSVERSION: 1400093
>> >>>> Started Sun, 09 Jul 2023 03:11:16 GMT 25:00:07 GMT and built over
>> >>>> 1500 ports in under 26 hrs:
>> >>>> ampere2 main-arm64 default
>> >>>> Host OSVERSION: 1400089
>> >>>> Jail OSVERSION: 1400093
>> >>>> ===
>> >>>> Mark Millard
>> >>>> marklmi at yahoo.com
>> >>>
>> >>> Since this does not seem to be mentioned in the previous posts. The
>> >>> phenomenon does not only occur in the pkg clusters.
>> >>>
>> >>> In fact, I also observe this severely delayed behavior under
>> >>> Poudriere on my very current 14.0-CURRENT box. All *-depends sections
>> >>> need a multiple of the usual times ...
>> >>>
>> >>
>> >> So, something like:
>> >>
>> >> Host OSVERSION: 1400093
>> >> Jail OSVERSION: 1400093
>> >> and:
>> >> /packages/All/pkg-1.20.1_1.pkg
>> >
>> > Yes, this corresponds exactly to my versions of the system, the jail and
>> > ports-mgmt/pkg.
>>
>> The newest update of ports-mgmt/pkg to version 1.20.2 by bapt@ solves
>> the problem for me :)
>>
>
> Great! So, can anyone restart the build or kill it until it will start
> again at 1:00 UTC tomorrow morning or will it start building with the
> updated pkg? I see no signs of using any change to this point.

Even with pkg-1.20.2 many running jobs are stuck with *-depends in Status.

https://pkg-status.freebsd.org/beefy18/build.html?mastername=main-amd64-default&build=pd6fe15e0ee7a_s1ec7cb1b5d
https://pkg-status.freebsd.org/beefy17/build.html?mastername=main-i386-default&build=pd6fe15e0ee7a_s1ec7cb1b5d
https://pkg-status.freebsd.org/beefy16/build.html?mastername=131amd64-default&build=d6fe15e0ee7a
https://pkg-status.freebsd.org/ampere3/build.html?mastername=131arm64-default&build=d6fe15e0ee7a
https://pkg-status.freebsd.org/beefy15/build.html?mastername=131i386-default&build=d6fe15e0ee7a
https://pkg-status.freebsd.org/beefy6/build.html?mastername=124amd64-default&build=d6fe15e0ee7a
https://pkg-status.freebsd.org/beefy5/build.html?mastername=124i386-default&build=d6fe15e0ee7a