Re: pkg server for current/arm64 stopped ? [main-armv7 on ampere2, elapsed so far: 651:21:56]

From: Mark Millard <marklmi_at_yahoo.com>
Date: Sat, 27 Apr 2024 03:21:54 UTC
On Apr 26, 2024, at 18:55, Philip Paeps <philip@freebsd.org> wrote:

> On 2024-04-18 23:02:30 (+0800), Mark Millard wrote:
>> void <void_at_f-m.fm> wrote on
>> Date: Thu, 18 Apr 2024 14:08:36 UTC :
>> 
>>> Not sure where to post this..
>>> 
>>> The last bulk build for arm64 appears to have happened around
>>> mid-March on ampere2. Is it broken?
>> 
>> main-armv7 building is broken and the last completed build
>> was the one started on Mon, 19 Feb 2024 12:32:10 GMT. It
>> gets stuck making no progress until manually forced to stop,
>> which leads to huge elapsed times for the incomplete builds:
>> 
>> pd5512ae7b8c6_s75464941dc 34472 12282  (+9196) 107  (+77) 4753  (+2247) 1390  (+529) 15940 parallel_build: Fri, 22 Mar 2024 11:05:01 GMT 651:21:56
>> 
>> p43e3af5f5763_sf5f08e41aa 19809 5919  (+3126) 137  (+100) 5363  (+2741) 1395  (+522) 6995 parallel_build: Wed, 28 Feb 2024 15:46:14 GMT 359:42:14 ampere2
>> 
>> ampere2 alternates between trying to build main-arm64 and main-armv7, so main-armv7 being stuck blocks main-arm64 from building.
>> 
>> One can see that all 13 job ID's show over 570 hours:
>> 
>> http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=pd5512ae7b8c6_s75464941dc
>> 
>> It is not random which packages are building when this happens. Compare:
>> 
>> http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=p43e3af5f5763_sf5f08e41aa
>> 
>> By contrast, the 19 Feb 2024 from-scratch (full) build worked:
>> 
>> http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=pe9c9c73181b5_sbd45bbe440
>> 
>> My guess is that FreeBSD has something that broken after bd45bbe440
>> that was broken as of f5f08e41aa and was still broken at 75464941dc .
> 
> It looks like ampere2 is going to end up in this state again:
> 
> https://pkg-status.freebsd.org/ampere2/build.html?mastername=main-armv7-default&build=p1c7a816cd0ad_s1bd4f769ca
> 
> It's got a couple of things stuck in -depends already.  I'll keep an eye on it for the next hour or two.  If no progress is made, I'll kill this build and force an upgrade.  The next build will start at 01:01 UTC Sunday.  So we won't have long to wait before it tries again.
> 
> ampere1 is chewing away at llvm, and doesn't look stuck.
> 
> ampere3 has been upgraded.

Output from the likes of:

# ps -axldww

could be interesting. As might be output from:

# pstat -k -k PIDs_OF_STUCK_PROCESSES

(kernel stack backtraces).


===
Mark Millard
marklmi at yahoo.com