From nobody Tue Oct 24 19:37:18 2023 X-Original-To: ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SFMlV0NMHz4yPx6 for ; Tue, 24 Oct 2023 19:37:26 +0000 (UTC) (envelope-from sysadmin.lists@mailfence.com) Received: from wilbur.contactoffice.com (wilbur.contactoffice.com [212.3.242.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SFMlT5LMvz4SS3; Tue, 24 Oct 2023 19:37:25 +0000 (UTC) (envelope-from sysadmin.lists@mailfence.com) Authentication-Results: mx1.freebsd.org; none Received: from ichabod.co-bxl (ichabod.co-bxl [10.2.0.36]) by wilbur.contactoffice.com (Postfix) with ESMTP id 4D493E05; Tue, 24 Oct 2023 21:37:22 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1698176242; s=20210208-e7xh; d=mailfence.com; i=sysadmin.lists@mailfence.com; h=Date:From:To:Message-ID:In-Reply-To:References:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc; l=6964; bh=QKx1iKml3Gd5FAh5/cVLY004v8qmpQ1OA0/J1SB63rU=; b=ir2QcehRNZ7CuT3Fj7huNtpXLpTArhVmjq7CnN7dvBheQH8m8jrCE4lT9P+uYoef IBOlRUJXhtYvqbZjvy61D/nC7rd4tzCpgIMvjdzoiLANSYv8EHjFRrb2HtgySWGRbsc /9DD813JIY+51Y3dELguq38enPXnmEYA1kuKv0Vq+LtA3JDWfmggDL08/R6ypp9AfQK hMeNtyjCjV33W5aUnSvhKkpLHpyDITyzIl3RwO4AaVhK02GAh4jVdQksJ24TjC9/8KL hOlwM526q+6ZvLMDF2Ad5Q+o4Vk6QL6fByxEDfuFhuu+bSHt1kZwqWPdQ+Cz9WZ99z9 SQhtwypNQg== Date: Tue, 24 Oct 2023 21:37:18 +0200 (CEST) From: Sysadmin Lists To: ports@freebsd.org Message-ID: <1781920819.96097.1698176238817@ichabod.co-bxl> In-Reply-To: References: Subject: Re: We need to do something about build times List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Robert Clausecker X-Mailer: ContactOffice Mail X-ContactOffice-Account: com:312482426 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:203, ipnet:212.3.242.64/26, country:US] X-Rspamd-Queue-Id: 4SFMlT5LMvz4SS3 > ---------------------------------------- > From: Robert Clausecker > Date: Oct 24, 2023, 12:12:13 PM > To: > Subject: We need to do something about build times > > > The build times have gone up to the point where they are unsustainable. > Frequent updates to key ports (like llvm*, rust, gcc*) make it so that > basically every time I prepare a new batch of commits, I have to rebuild > a variety of toolchain ports across 8 jails (amd64/i386/arm64/armv7 each > for FreeBSD 12.4 and 13.2). This takes multiple days. And I'm working > with hardware that's quite recent (for x86, an 8 thread Skylake box, for > arm, an 8 thread Windows 2023 dev kit). > > By the time the builds are done, some random update has usually caused > the ports to be out of date again, so if I were to rebase, I would have > to do all of this again. And again. And again. > > Particularly bad offenders are gcc and rust. Ccache is ineffective for > these as gcc has LTO turned on, which seems to more than triple the > regular build time to more than 24 hours even on a fast Skylake box. > This is single threaded as I build multiple ports at once; if I were to > build multi-threaded, the same amount of total CPU hours would have been > spent, so that would not fix my problem. Ccache is also ineffective for > rust of course. > > There's another issue in that ccache doesn't scale to large cache sizes > (my experiments show that anything larger than 20 GB seems to cause > problems as ccache repeatedly tries to scan the whole thing for evictions), > and the sizes that work are just not enough to be effective. What would > help is being able to have one cache for each combination of ports tree > and jail, but Poudriere has no support for that. > > Another bad offender is texlive. For some reason, texlive-texmf needs to > be rebuilt frequently, despite mostly comprising data that is just > unpacked and repacked. This takes forever and pegs the disk at 100% for > more than an hour as the texlive source tarball is repeatedly extracted > and then compressed into packages. I don't get why the texlive stuff is > not split in such a way that the stuff that is just repacked lives in its > own port with no dependencies so it only needs to be rebuilt on rare > texlive updates. > > And it seems I'm slowly killing my build SSD like that. After just about > 9 months, it is already at 100 TB of writes just from port builds. > Building with workdirs in memory is no longer an option as that frequently > kills my build server by filling all its RAM with build files until no > processes can be started anymore. Poudriere does not have an effective > mechanism to prevent this (tmpfs limits don't work as the ports in > question require very large workdirs, tend to take very long to build and > tend to be built all at the same time for multiple jails). > > Using prebuilt packages is not an option as they lag behind by several > days/weeks and lead to an inconsistent testing environment. It is also > not a good solution to chose non-default build options for these ports > as it is not clear if that would affect the validity of the testbuilds. > > How can we fix this problem and make ports development sustainable again? > > Some ideas: > > - disable LTO and other options by default that increase build times by > such a ridiculous degree. This would really make a huge impact with > very little work. I don't think LTO on toolchain ports improves build > times enough in comparison to the extra time it takes to build these. > > - for gcc, switch to single or no bootstrap by default. We have known > good toolchains we use to build gcc. There's really no reason to > build it multiple times just out of paranoia. The maintainer is > supposed to check that gcc is built correctly without bootstrapping > so consumers don't need to build it multiple times. > > - untangle some of the dependencies so that less ports may trigger > rebuilds of critical ports. For example, llvm docs could be moved to > separate ports so that updates in the documentation toolchain do not > trigger an LLVM rebuild. > > - reduce USES to chose lighter dependencies by default. E.g. USES=llvm > could depend on the light flavour by default. I'm sure only very few > ports need all of LLVM and the light flavour is faster to build. > > - rework Poudriere's rebuild detection to not rebuild every port for > every random bullshit thing. For example, I don't see why ports need > to be rebuilt for transitive changes in build dependencies. E.g. if > port A has build depends on port B which build depends on port C, and > C is updated, then A has to be rebuilt despite its direct dependencies > being unchanged. This does not appear to be reasonable. > > - unbundle libraries more thoroughly. We currently have dozens of > copies of LLVM, skia, webkit, and others in tree as ports just bundle > them instead of even making an attempt at unbundling. This means that > every time they need to be patched, it's a whackamole at finding all > copies. Plus build times suffer a lot. I know it's hard, but perhaps > something can be done. For example, I have given up on trying to make > electron work on armv7 as with every major version update, my patches > are randomly being dropped and I have to do it all again. Like all > chromium ports, electron takes over two days to build on my arm box > and my time is insufficient for that. > > - stop bulk bumping RUN_DEPENDS consumers when dependencies are updated, > or at least think carefully before doing so. RUN_DEPENDS are only > installed after the build and should not affect the build. For > example, sysutils/cdrtools uses the command line opus encoder and thus > depends on audio/opus. There is absolutely no reason to bump it when > audio/opus is updated. It just causes everybody to needlessly rebuild > and reinstall ports. Sure there's the odd case where that needs to be > done, but it seems like some maintainers just always do that, even > when it's not needed. > > - maybe add a system where ports can declare the oldest version of > themselves they are compatible to, in the sense that consumers only > need to be rebuilt if they were built against a version older than > that. For example, if a shared library is updated with a bug fix > that does not change the ABI, there's no need to rebuild all consumers. > > With great frustration, > Robert Clausecker > > -- > () ascii ribbon campaign - for an 8-bit clean world > /\ - against html email - against proprietary attachments Well done diagnostics reporting, and the suggestions look reasonable to me as well. I suspect we're all getting frustrated with ever-increasing build-times, and now's a good time to address their underlying causes. -- Sent with https://mailfence.com Secure and private email