From nobody Sun Jun 30 18:48:26 2024 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WByqk4hZCz5QGGX for ; Sun, 30 Jun 2024 18:48:34 +0000 (UTC) (envelope-from mirror176@hotmail.com) Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04olkn20800.outbound.protection.outlook.com [IPv6:2a01:111:f403:2c09::800]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WByqj64GTz4bHl for ; Sun, 30 Jun 2024 18:48:33 +0000 (UTC) (envelope-from mirror176@hotmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hotmail.com header.s=selector1 header.b=aKUbYIyl; dmarc=pass (policy=none) header.from=hotmail.com; spf=pass (mx1.freebsd.org: domain of mirror176@hotmail.com designates 2a01:111:f403:2c09::800 as permitted sender) smtp.mailfrom=mirror176@hotmail.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FiqZm9uScK2Zr8Z2iONAI37KkhKVxtnbu/CjNuelYriNcJwm/lBFu2Cd2Z195yHYIFxgbQzlBXpAm1mKYT0UJYkG3pLnWInjNy16M+2qUMrk4faK6SUbKLi9WScplssdP3tTPHD0RNR2k9cT2SJ6f1m1ej44XbwMCLMbEhuMNiAgt3cvAjuwpwboMgjVC8Dsds8A/cv8wUyqQRehtVTlBY6OgoBNI33zI4kdxs6daoWCtbjMHSWFAvkZavI7L+Y2fJHet2uEDJzXXlZthwHlRa3E4SiCDi9usnX6cFTaaDb35DWyWTHmIvayuRtg2zqF4F4eIJEpTgPJJgwJFiPjgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dLSog8VA2bTEGobo6q9edskhNDxyDJ/K4/Cv1VFPOEo=; b=mYzEilMK5EyS1Udgo2jSLJFH8BqS0uEVOJOKMkDdSi927GYGQyUwp5kv1bTuU1Ddtq28y+YDbL3vdTNH77Bz8CJwNGqoZhg520Haw4l/4c3mw2KrkTtjd9Q4z285q3osSEycIHfyRaGqjmQnNhy/Hu3Z5mPc86ELz5f+0FkOgOKsLBGeH8u7PffcDArGDys4ZaGhhd2ZXh9xHRLZ74swCccnvoW3LA48pY4ciqXKrHOTH7RgnUY42hTXhw43ENKQzYegkWCGFT8WUbfUf/OpKWU+FVA1mP78rOmFsxiasUiO73gC1IliYhkreKprRls6X+3y4zhMyvcckBPoDYgAig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dLSog8VA2bTEGobo6q9edskhNDxyDJ/K4/Cv1VFPOEo=; b=aKUbYIylfH0pvOpq9G/zyTBCTTHFcc70sRMoecVlnz/pSOdjOARn3pwlfNQAjPDKioh9v2/DEGMhDQQi/iyPr19hjyCLqMAoYzIHg8sZRVU2eAUz1DSS5zq5opg1jNPgbMl76rqHLSkxpVr19wAw5i1sXf2A/Zmw24DWuhhNEjXXa5yRaWT+EwPeel0h3fIJeM1Uje/5ilGIhuuWcK6AbrWS7dhxXTJnGrvVvFcu5bvurVK5TFxz0VADWBe0tvUiV+mjUjdk+pc3q/u8It9OBBU8g9Mu45fN/XdleDZ1dMb5O06OKFqXe0Qg+qoUExVZ7ao26BK8KYCyN71ind51HQ== Received: from CO1PR11MB4770.namprd11.prod.outlook.com (2603:10b6:303:94::19) by CY5PR11MB6341.namprd11.prod.outlook.com (2603:10b6:930:3e::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7698.34; Sun, 30 Jun 2024 18:48:30 +0000 Received: from CO1PR11MB4770.namprd11.prod.outlook.com ([fe80::bffd:9e35:4afa:a747]) by CO1PR11MB4770.namprd11.prod.outlook.com ([fe80::bffd:9e35:4afa:a747%4]) with mapi id 15.20.7719.028; Sun, 30 Jun 2024 18:48:30 +0000 Message-ID: Date: Sun, 30 Jun 2024 11:48:26 -0700 User-Agent: Mozilla Thunderbird Subject: Re: Buildworld Taking Very Long Time To: questions@freebsd.org References: <3ca5191f-41fe-420c-86e9-ee4e89e599a9@gmail.com> Content-Language: en-US From: "Edward Sanford Sutton, III" In-Reply-To: <3ca5191f-41fe-420c-86e9-ee4e89e599a9@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-TMN: [07FIISQDQT/lO5wwteW/emDXU6sV8aAH] X-ClientProxiedBy: BL1PR13CA0376.namprd13.prod.outlook.com (2603:10b6:208:2c0::21) To CO1PR11MB4770.namprd11.prod.outlook.com (2603:10b6:303:94::19) X-Microsoft-Original-Message-ID: <21a74e22-b4f0-4080-aec4-659026735584@hotmail.com> List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-questions@freebsd.org Sender: owner-freebsd-questions@FreeBSD.org MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PR11MB4770:EE_|CY5PR11MB6341:EE_ X-MS-Office365-Filtering-Correlation-Id: 4d8ad195-e194-4b9b-9dbc-08dc99353b9a X-Microsoft-Antispam: BCL:0;ARA:14566002|8060799006|461199028|3412199025|440099028; X-Microsoft-Antispam-Message-Info: Dxs42d6lQpdciCm9tPvJ3YP88xEFx59HtChFevufFN5X4UUB4Erduj1ugkKHiafGm2jxWIKZEIekjI+9QCeAyfIz0Zdnq3mA6CM13L6HKmsZ4US3yZv26e+UQbw30dsnk1PNPxFCw93YDKPry5bQgn4FmwArBZXq6+6QmsJQhpWc5fw3WQ6+/6TOtY1836jHQKGDuWjnDklR6bNCDTVRxHbyXwpMhTMzT4n9m0an+Cp/GbOv8VqDpeXMLhcpeeXjQ/CnC8oijsZ82/+zvVLzje3GjBohXy5xUziWVfgMILyvN0GgJqgxLrUwHlA29wr5m+5uAQSvKBPACSrOHNwaPmHnPxCoTt8Mvw4PXsyDznFYwP+BC3ucULKKXdFjwZr+hDvO6x1FJ6IQeald3TYvyEOo2yWhu/cNdgZ6OZpf4egTyM19Ce9xDVVfrt6uUdbKo92P6Enq+4xm/lyyXEg7xeM+Ld0X+LgPrmS3hUszdihdCNvmaMfLYwx1KtKdroL8DrwAAHV5Ugu4F3UGHpKzH2ApIzoVgRBCnqiCP1B8EBsveNKAn5vvwM8FELN3lOqBi6qjIhiMKeD00rhZOa+wv5UXszl64SCf61WCORGtNlps+FfNoC15uiYRmLt+1S+j X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OTdRanJha1dET2FVYXdLbVM3dVVxVFo5V1VSblBMVFI0MUpSdEltS3MvVmRB?= =?utf-8?B?ZEdIUXRIRnNscVFkSkhSYjJJRGpTVzV2LzZOVHlkeTBWeWQ3Nzk1aW9xWDJB?= =?utf-8?B?S0hmbHR6U3o2L081NXJFTG1pMEViUHFwNjY1Vmx0QVhxU20xMkJielFEalNq?= =?utf-8?B?NFpVZFBXb2wvOCtSWVBMNUE0U1BlL3F3Wk54T0hKRGFXOGZKdlpzY1I2QTJY?= =?utf-8?B?ZmZQekdPajZGMW5aaDAxZjBocU41dVRXYmVxWFFka1JtR3MwSktXT2hNUjlk?= =?utf-8?B?c3ljVllieFlNSXNyZHg4d3BKak9PaDZaVWtEbGUzZkpEdUlyWTFzSUhCK25q?= =?utf-8?B?ZXVlQWRkN0t6bVE2K1BJZFZNWkNKN2UxeFdyN3VsckVPTUpYVllOS1BNV2FD?= =?utf-8?B?K3pId3UyU1VJckdod2dpQldpVTdPdTJ3ZjBtNFRzUkFJUytGcWRYUFphdmd1?= =?utf-8?B?TnpRWVhQTjA3dnBTamZjeDE5MEVITTd2SG1ieUlrWVI2Ykc4amhkSlB5SEY0?= =?utf-8?B?ek5VYlBjUVo4V25HdXZ1dUdpMHdqaDF2RmkrcnIzWlgzU0RHejZlY2NPTmJh?= =?utf-8?B?UHk4TGNKRlBNQUNUMkFaVjhLSjJuSmVxemhnRFBXcHhTam5BQmU1eEc0b3FO?= =?utf-8?B?Q1dGekpmNXgxMVpPMjJlTktTdEpRYWNmTDFuTkx2TDVOd2hiSGlxZmhoOFNW?= =?utf-8?B?ZStHSGQxTlpQdWxiRHAxNXplQjBPRUxkblVYUjRsWUZkb3ZDQXQwQVNhODU4?= =?utf-8?B?YmxiREpTNXl4NUJzRDJwdmVndCtQRzJvT2Jkc3NCL2c0OUlMaVZyZmxqVGhC?= =?utf-8?B?bkcvRzFtNmRtdTc2ZTl4Unc1RVZlSHZkNDU1cjI5OUsxbjRMWEMvZytjbjVB?= =?utf-8?B?N2VKNGlSYSt5TFhuNHJ5OU5jUyt3V1lpUkw4Mm5hUC9DZFdOa25yMnNrbUxu?= =?utf-8?B?aS9OSEtOdk9rdWdDeTFEZXBvMjlsZ243cnEvL1BGSVdmbE1xTGdjOThrOGFG?= =?utf-8?B?YTFGR3lHV2FGQkU3Tng1alRHcnMzY0VZNzY5WTltNE14WDBKVXRVNHlWV1FC?= =?utf-8?B?MzhZOVRaOGV5VGpOTjFPcXk4ZHk0SWNxalhTeGZBSEtVOHJrU3U2QmUrMm5M?= =?utf-8?B?bnZrTDR0N2t5SXcxbWRlRnB6dXNySmlvQ1puNnNia3lmeHlXM0x2ZVRXOTlr?= =?utf-8?B?Y21ZT2pJaitHbStZK29OSkY0SmhPVWxJRVlwcTUza3NpZ0lud0xlY0VaU0pH?= =?utf-8?B?VGdqVGd4cGFnUTJpWE9lK3J2WkRRRTNka1NYUEZBRWV3SU9CejdNVXI2aUM4?= =?utf-8?B?d1lWL0gyYWZrUjVyazF6VEdnblB0OGJjL3BnWjZzNkIvRlFNMjFJTlVXeHQw?= =?utf-8?B?NGJqTWNmcHJwUVNoRkVTVXY4TkxmcGl1QnlUZXZ0a2R1b0pvQVpRRndZK3Zm?= =?utf-8?B?SE9waUk0TmpQQ1FPUldKMUNqSEZrczUvd2ZZU0huKzlZK3p1Y0RyajUrZWl1?= =?utf-8?B?bldHOThSc1Z2ZWNkQzhkbGxITG1NMVBLOGlGMEN0VzJjMmZhU3g2TEFNa1o1?= =?utf-8?B?Tkd1Tnc3aGtaZlo4M3RLU3JLWjhJV3ZkUm1iZWhRQzNlTnZFZEp5MW1La3RT?= =?utf-8?Q?ZExlaBRrCv/d/dw5MLPm8UPrF7q1n3hS0AsWnEzrgL48=3D?= X-OriginatorOrg: sct-15-20-7719-20-msonline-outlook-1ce3f.templateTenant X-MS-Exchange-CrossTenant-Network-Message-Id: 4d8ad195-e194-4b9b-9dbc-08dc99353b9a X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4770.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Jun 2024 18:48:30.7229 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR11MB6341 X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.45 / 15.00]; FORGED_MUA_THUNDERBIRD_MSGID_UNKNOWN(2.50)[]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.96)[-0.965]; DMARC_POLICY_ALLOW(-0.50)[hotmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f403::/49]; R_DKIM_ALLOW(-0.20)[hotmail.com:s=selector1]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; FREEMAIL_FROM(0.00)[hotmail.com]; MIME_TRACE(0.00)[0:+]; DWL_DNSWL_NONE(0.00)[hotmail.com:dkim]; FREEMAIL_ENVFROM(0.00)[hotmail.com]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[questions@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[hotmail.com:+] X-Rspamd-Queue-Id: 4WByqj64GTz4bHl On 6/30/24 08:11, Tim Daneliuk wrote: > We do a nightly pull of -STABLE and then a buildworld/buildkernel stable/14, stable/13, and/or no longer supported stable versions? > The world and kernel build typically has been taking about 45-60min on > one of > our quad core i5 machines. i5 narrows it down to 19(?) generations of CPUs. 4 core cuts it down to about 9. CPU performance/features can vary a lot across those generations+models. > For no obvious reason, it's now taking dozens of hours.  Any insight on > why this > might be happening would be appreciated. My system using meta mode and ccache for stable/14 if running a build attempt with filesystem data cached in RAM after a build completes causes it to rerun within minutes on my i7-3820 using only 6 cpu cores, 32GB RAM, and a single magnetic hard drive. Running an update when clang has been updated takes hours (not tens of hours) and I thought I recall a decent amount of time goes to openssl too. A full build after cleanup of the work directory should still be below 10 hours; my last timing was with 4 cores on the otherwise same hardware given a -j16 and took less than 6 hours ago but it was long enough ago I don't remember if it was timed during /13 or /12 (I delayed the 14 update for a while but may be new enough to be in that window but not of that build). Been a number of days since the last clang update in /14 but openssl did just get updated; still doesn't likely explain 1 hour to 1 day+ buildtime change. More build hardware+software setup is likely needed: Specific CPU, preferably RAM total+speed, what storage media (magnetic/ssd, models, array configuration if RAID. What filesystem is on the drives. Any build customizations (ccache, WITH_META_MODE, altered compiler flags, number of make jobs). What version of OS. If PORTS_MODULES is defined it can add additional complete compilers to the build process among other things from the ports tree depending on its state and the state of currently installed packages. Have you observed any unusual stats like lower CPU, higher disk I/O & % busy compared to a typical run? If you don't have specific stats you could glance at how things appear with top, systat, etc. to start getting an idea. Do you know what steps in the world/kernel are taking long? You can separate buildworkd and buildkernel into separate commands and time them separately. `make -s buildworld` will suppress a lot of output which helps see stages messages and the entire build can be logged. I don't know how but I imagine there is a way to do it with timestamps throughout. Using magnetic media, ZFS with compression, ccache, and leaving atime=on can lead to horrendous disk performance. I 'think' atime causes fragmentation of file metadata (even listing large directory contents takes forever) but even if not you still have 1 write for every file read; disabling it likely causes ccache to clear the cache as a first-in first-out sequence instead of removing what hasn't been used in the longest time. devel/ccache on a compressed dataset doesn't track sizes properly which sounded like zfs reports new cache entries are 0 bytes instead of returning its uncompressed size (compressed size can't be returned until compression algorithm is completed). This causes `ccache -s` cache size to exceed max cache size without triggering automatic cache cleanups; manually running `ccache -c` gets the cache back within limits which can make a much smaller cache and can have massive performance improvements if the file count was getting out of control. A very poorly performing ccache storage even reveals questionable calls to ccache from ports tree operations as basic non-compiling operations now become very slow with ccache disk I/O. I haven't had WITH_META_MODE cause a noticeable detriment to build times but have had it break builds until I ran `chflags -R noschg /usr/obj/usr;rm -rf /usr/obj/usr;cd /usr/src&&make cleandir&&make cleandir` though if trying to diagnose this for yourself and others it would be helpful if you moved/backup instead of removed the build directory contents so it could be further analyzed. Are there any other uses this machine has during build that could be hogging CPU/RAM/disk with other operations? Are CPU temperatures staying in proper range or could thermal throttling be ruining CPU performance? Disk I/O taking longer than expected on a filesystem with plenty of free space and reasonable file/directory count could indicate a drive issue; running SMART tests, reseating all drive cable connections (helps with dirt/minor corrosion; disconnect+connect several times), and making sure drive temperatures are within adequate ranges is good.