Re: pkg server for current/arm64 stopped ? [main-armv7 on ampere2, elapsed so far: 651:21:56]

On 2024-04-18 23:02:30 (+0800), Mark Millard wrote:

> void <void_at_f-m.fm> wrote on
> Date: Thu, 18 Apr 2024 14:08:36 UTC :
>
>> Not sure where to post this..
>>
>> The last bulk build for arm64 appears to have happened around
>> mid-March on ampere2. Is it broken?
>
> main-armv7 building is broken and the last completed build
> was the one started on Mon, 19 Feb 2024 12:32:10 GMT. It
> gets stuck making no progress until manually forced to stop,
> which leads to huge elapsed times for the incomplete builds:
>
> pd5512ae7b8c6_s75464941dc 34472 12282  (+9196) 107  (+77) 4753  
> (+2247) 1390  (+529) 15940 parallel_build: Fri, 22 Mar 2024 11:05:01 
> GMT 651:21:56
>
> p43e3af5f5763_sf5f08e41aa 19809 5919  (+3126) 137  (+100) 5363  
> (+2741) 1395  (+522) 6995 parallel_build: Wed, 28 Feb 2024 15:46:14 
> GMT 359:42:14 ampere2
>
> ampere2 alternates between trying to build main-arm64 and main-armv7, 
> so main-armv7 being stuck blocks main-arm64 from building.
>
> One can see that all 13 job ID's show over 570 hours:
>
> http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=pd5512ae7b8c6_s75464941dc
>
> It is not random which packages are building when this happens. 
> Compare:
>
> http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=p43e3af5f5763_sf5f08e41aa
>
> By contrast, the 19 Feb 2024 from-scratch (full) build worked:
>
> http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=pe9c9c73181b5_sbd45bbe440
>
> My guess is that FreeBSD has something that broken after bd45bbe440
> that was broken as of f5f08e41aa and was still broken at 75464941dc .

I'll kill the build on ampere2 again.  Thanks for the nudge.

We don't really have good monitoring for this.  Also: builds should time 
out after 36 hours.  The fact that this one does not is a bug in itself.

Philip [hat: clusteradm]