From nobody Thu May 09 00:28:55 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VZXv12zJBz5K3g3; Thu, 09 May 2024 00:29:01 +0000 (UTC) (envelope-from philip@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VZXv12NqVz4jxf; Thu, 9 May 2024 00:29:01 +0000 (UTC) (envelope-from philip@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1715214541; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=neNkO5Vu1HvdOYZChBpeUuTnngjIRjARQ5gAF43WPCE=; b=hHYdlENQn7pk3FuSAKV02P1SMyd/hRAzy+zEQ7eEXzR7wGExxAZLZEzd/5Bhm7ewrDe/MD zhqR3Ouofya9Zr3RD7R2bcma7zLqTFVXlTh+V0o/Ts+AC/d4bstkGlijcA8BD1mCaQAmfz edU14PHNJROt/g8/+HOUeLNdQVLV+xlDoOUWiUDDMxtHAzQ2erXcBGNuWDybpW2LUqkGnx ZcqEexxZVVeA8zrBzVmVNM8c829VPzeJyh2fR8OIT+hhYxqX2lnjTwxGoRUMSghDvEobQu ziWNUtnUM7cnn2ZueQBa9jElC2qktvaigRdIE6UH4mVePGuJzsCTG0KAjoXUug== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1715214541; a=rsa-sha256; cv=none; b=IbMx6L3sierAV9yDABBO7ACRp1ERFyvOs1gCLEPE51IM/tMtoVwEJ6Z6IzE324/rAgAETA eOf1PQTGws8vQRwv8FTRB7ZDBRCdxFM3HptctipdPob7B/UiWzJfN/udRL4Io9kvCWHfB/ 1hLydrN9++lI3Zi5zoYY66sV8h5FXbo6K9uhXGtCUY34gb0+kUYpYK4qAszIQ7wlZyGLE1 bl6wYFDkq/E2wAntnWs3KM/K0MEw33/7XtXtAmCeSAxESxqy9dnQGm6X0iJshc3chk3Irz 8G0t9GVdXItwJC+RoCo1IMs+LwFCSJ9EarsaSTBjEb+nEVl7pvMp7yS861PhUg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1715214541; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=neNkO5Vu1HvdOYZChBpeUuTnngjIRjARQ5gAF43WPCE=; b=MJpH8TwxAupUqU6IKm7LCLc63RHorEcXSklwuuLu6rQh4dr1oo9TpedSKUXu9oNY5281CZ BC9QQP882e2ZQj2K32QCkMcVaW7KEiNSjxJgAGG6vAG2L4lb0cSwQiwKD+x5cDRhLg9ZFe qOPTyjGvq7ZS8pSZxXkiRZG29lg5i9QvDLDni4vlJzTWL7SL4nShUOsUyPSyVfjAnTWtB5 2345Bd9r4lebnBlpNO0kEu52reE4M1sNL9eZU9WGwPQspGexVS2orsLWSdeXQsfis5oFXv 84QRaTp6Yn4LLqLJahn9HH9fiBi5qmb4sCCnO5q3mw2a08DaWaBd5aSO37ByAQ== Received: from fauth1-smtp.messagingengine.com (fauth1-smtp.messagingengine.com [103.168.172.200]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: philip/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4VZXv111JMz1SVN; Thu, 9 May 2024 00:29:01 +0000 (UTC) (envelope-from philip@freebsd.org) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfauth.nyi.internal (Postfix) with ESMTP id 429171200066; Wed, 8 May 2024 20:29:00 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Wed, 08 May 2024 20:29:00 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefuddgfeegucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffokfgjfhggtgfgsehtqhhmtdertddtnecuhfhrohhmpefrhhhi lhhiphcurfgrvghpshcuoehphhhilhhiphesfhhrvggvsghsugdrohhrgheqnecuggftrf grthhtvghrnhepgefhveefgeehtefhieevtedulefftdejheefgffgfffhgfelvdeihfdu iedutdelnecuffhomhgrihhnpehskhhiphhpvggurdhgrhgrphhhihgtshdpfhhrvggvsg hsugdrohhrghenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhr ohhmpehphhhilhhiphdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqudduie eivdeivdegkedqvdefhedukedttdekqdhphhhilhhipheppehfrhgvvggsshgurdhorhhg sehtrhhouhgslhgvrdhish X-ME-Proxy: Feedback-ID: ia691475d:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 8 May 2024 20:28:58 -0400 (EDT) From: Philip Paeps To: Mark Millard Cc: void , FreeBSD Mailing List , Current FreeBSD Subject: Re: pkg server for current/arm64 stopped ? [main-armv7 on ampere2, . . .] [Update to Host OSVERSION 1500018 did not help] Date: Thu, 09 May 2024 08:28:55 +0800 X-Mailer: MailMate (1.14r6030) Message-ID: <11DB2E73-A765-47CD-9C5A-B4F8C6978704@freebsd.org> In-Reply-To: <8B799B49-C5BD-428B-AEFA-042E1371BF4D@yahoo.com> References: <03736C90-EE54-47B3-AEA7-ED1AC0343B4B@yahoo.com> <046DE453-9F01-4D07-9E14-EF22E537A163@yahoo.com> <5243EC66-2A75-4C06-B2F7-E9E6E6C2840A@yahoo.com> <3CF25092-E285-43E2-9B7A-F5108E4E0A7A@yahoo.com> <8B799B49-C5BD-428B-AEFA-042E1371BF4D@yahoo.com> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; format=flowed Content-Transfer-Encoding: quoted-printable On 2024-05-08 23:53:57 (+0800), Mark Millard wrote: > On Apr 29, 2024, at 20:16, Mark Millard wrote: > >> On Apr 29, 2024, at 20:11, Mark Millard wrote: >> >>> On Apr 29, 2024, at 19:54, Mark Millard wrote: >>> >>>> On Apr 28, 2024, at 18:06, Philip Paeps wrote: >>>> >>>>> On 2024-04-18 23:14:22 (+0800), Mark Millard wrote: >>>>>> On Apr 18, 2024, at 08:02, Mark Millard = >>>>>> wrote: >>>>>>> void wrote on >>>>>>> Date: Thu, 18 Apr 2024 14:08:36 UTC : >>>>>>> >>>>>>>> Not sure where to post this.. >>>>>>>> >>>>>>>> The last bulk build for arm64 appears to have happened around >>>>>>>> mid-March on ampere2. Is it broken? >>>>>>> >>>>>>> main-armv7 building is broken and the last completed build >>>>>>> was the one started on Mon, 19 Feb 2024 12:32:10 GMT. It >>>>>>> gets stuck making no progress until manually forced to stop, >>>>>>> which leads to huge elapsed times for the incomplete builds: >>>>>>> >>>>>>> [...] >>>>>>> >>>>>>> My guess is that FreeBSD has something that broken after = >>>>>>> bd45bbe440 >>>>>>> that was broken as of f5f08e41aa and was still broken at = >>>>>>> 75464941dc . >>>>>>> >>>>>> >>>>>> One thing of possible note: >>>>>> >>>>>> Failing . . . >>>>>> >>>>>> Host OSVERSION: 1500006 >>>>>> Jail OSVERSION: 1500014 >>>>> >>>>> I have finished a package builder refresh this morning. All our = >>>>> builder hosts (except PowerPC - I don't touch those) are now on = >>>>> main-n269671-feabaf8d5389 (OSVERSION 1500018). >>>>> >>>>> ampere1 successfully finished its 140releng-armv7-quarterly build, = >>>>> so it looks like the problem with stuck builds was limited to = >>>>> ampere2 building main-armv7. I'll keep a close eye on this one = >>>>> when it starts its next build. >>>>> >>>> >>>> I see that main-armv7 started. >>>> >>>> It queued only 31935 instead of the prior 34528 (or more): it is = >>>> doing an >>>> incremental build instead of a full build. For example, pkg was not = >>>> built >>>> but instead the prior build is in use. Thus bad results from the = >>>> prior >>>> build might be involved in this new build. >>>> >>>> I'd recommend forcing a full "poudriere bulk -c -a" that does a = >>>> from-scratch >>>> build for the purposes of the main-armv7 test. >>> >>> Actually the test is not going to previde the information we are >>> after as things are. >>> >>> giflib-5.2.2 failed to build, which leads to devel/doxygen being >>> skipped. devel/doxygen was the first one to hang up in the prior >>> 2 failing attempts, if I remember right. >>> >>> giflib-5.2.2 also causes graphics/graphviz to be skipped. >>> graphics/graphviz was installed just before the hangup in all of >>> the example hanups. So the context will not be replicated. >>> >>> We need graphics/giflib to build to actually do the test. >> >> Looks like: >> >> https://cgit.freebsd.org/ports/commit/graphics/giflib?id=3D5007109903f= c271e3ef0ba01d78781c1fed99f3f >> >> is the fix for the graphic/giflib build failure. > > Well, main-armv7 is building again and things are still > getting stuck. So much for my idea. For reference I > list the over 10-hr-so-far ones: > > doxygen-1.9.6_1,2 build-depends 13:03:54 > py39-pydot-2.0.0 run-depends 12:24:04 > py39-pygraphviz-1.6 lib-depends 12:10:38 > > "ps -alxdww" would likely be appropriate to get a copy > of the otuput of. > > "procstat -k -k" usage and the like on stuck processes > would probably be appropriate. > > Does anyone with appropriate investigative background > have login access to ampere2 to take a look at what > is getting stuck? This is unfortunate. I'm sure I have the appropriate background, but = I'm spread very thin! I'll get as much information as I can about this = machine while it's stuck, before I bounce it again. I think it may be worth a try building those ports in isolation on = ref14-aarch64, and see what they're trying to do. I'll also set up a = set of refX-armv7 jails on that machine. Hopefully we can get to the bottom of this soon. This is a very tedious = failure mode. We could also try to put an older armv7 image on the builder jail on = ampere2. Depending on whether we have a sufficiently old image, that = will either be very straightforward, or a very deep rabbit hole. Thanks again for keeping an eye on this. We really should have better = monitoring for stuck builds than "Mark will tell us". :-) Philip