From nobody Thu May 02 20:18:09 2024 X-Original-To: ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VVlpy4cHfz5K0S0 for ; Thu, 2 May 2024 20:27:22 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VVlpy0ZN7z3x9V; Thu, 2 May 2024 20:27:21 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.18.1/8.18.1) with ESMTPS id 442KR6fF030049 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 2 May 2024 22:27:06 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) ARC-Seal: i=1; a=rsa-sha256; d=uucp.dinoex.org; s=M20221114; t=1714681629; cv=none; b=rhOLiKU3pSPfXfwgEYiJwSoBgb0Ugg8atOtG0HsGUvymCycYFmQfpGuvfE6EOgdZ1m4KlhZyT88lJfhPCpn56vQKoYqC/xDDFFGypm1dpnGvVDNHrHWAbxu04b5OpQtMuoioEnNOSaNu8pkPq76c9mcr3DOXPrHBzCW18CczB24= ARC-Message-Signature: i=1; a=rsa-sha256; d=uucp.dinoex.org; s=M20221114; t=1714681629; c=relaxed/simple; bh=fGdDhtWtgkk3NZzkHJV++eIveAxfHIxD7+Bd9WGnn7Q=; h=Received:Received:Received:Received:X-Authentication-Warning:Date: From:To:Cc:Subject:Message-ID:References:MIME-Version:Content-Type: Content-Disposition:In-Reply-To:X-Milter:X-Greylist; b=ldGCV8U4bq9dhu1wpmJC3X9acFGIcHg6R6c3g5DCw+GrEVDHUSZTgjrBu5Pu60TBU4XObjPdxcFqHcgKNQlDTRVoFr612guV/jAf6QGtaTtpP5bMrA2lj0ulEwi+98wpz1PHy5UN7aYD4Z1+dI8IdO+RMJ+GmlOQlIqHdmgcGG4= ARC-Authentication-Results: i=1; uucp.dinoex.org Received: (from uucp@localhost) by uucp.dinoex.org (8.18.1/8.18.1/Submit) with UUCP id 442KR6FS030048; Thu, 2 May 2024 22:27:06 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.18.1/8.18.1) with ESMTPS id 442KKCfq001562 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Thu, 2 May 2024 22:20:12 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.18.1/8.18.1) with ESMTPS id 442KI99H032379 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 2 May 2024 22:18:09 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.18.1/8.18.1/Submit) id 442KI9fi032378; Thu, 2 May 2024 22:18:09 +0200 (CEST) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Thu, 2 May 2024 22:18:09 +0200 From: Peter To: Brooks Davis Cc: ports@freebsd.org Subject: Re: git ports-tree doesn't work, please check! Message-ID: References: List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-ports@freebsd.org Sender: owner-freebsd-ports@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Thu, 02 May 2024 22:27:09 +0200 (CEST) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE] X-Rspamd-Queue-Id: 4VVlpy0ZN7z3x9V On Thu, May 02, 2024 at 07:08:02PM +0000, Brooks Davis wrote: ! On Thu, May 02, 2024 at 08:39:24PM +0200, Peter wrote: ! > ! > $ fetch https://cgit.freebsd.org/ports/log/devel/py-python-dateutil ! > fetch: transfer timed out ! > ! > My local replica has the same problem, git goes into a loop at that ! > location. ! ! Hmm, testing locally there's only a single commit to that path (moving ! it from Move devel/py-dateutil) so git has to read the whole commit ! history to discover there is only one commit. Even on a 40-core machine ! with nvme disks that's very slow process (but it does eventually ! complete). There is only that single commit, yes. And git has to read the whole history to find that there are no more. But that is true for any location if you read the log all to the end. # time sh -c "git log -- bin/bash > /dev/null" 10.199u 1.824s 0:16.38 73.3% 3413+446k 4619+0io 6829pf+0w Again, without spinup (these are mechanical) and from the ARC: # time sh -c "git log -- bin/bash > /dev/null" 5.446u 0.456s 0:05.91 99.6% 3409+446k 0+0io 0pf+0w Now a tough one: # time sh -c "git log -- www/firefox > /dev/null" 14.071u 0.707s 0:16.74 88.2% 3382+442k 1270+0io 1586pf+0w But then: # time sh -c "git log -- devel/py-python-dateutil > /dev/null" 82.180u 0.598s 1:23.64 98.9% 3418+447k 935+0io 1630pf+0w And it gets worse on repetition: # time sh -c "git log -- devel/py-python-dateutil > /dev/null" 113.464u 0.331s 1:55.99 98.1% 3411+446k 0+0io 0pf+0w # time sh -c "git log -- devel/py-python-dateutil > /dev/null" 133.253u 0.441s 2:22.12 94.0% 3413+446k 0+0io 0pf+0w This is not the io and pf, it is a compute loop in git - and it is done single-core, so neither vCores nor NVMe help here. ! I suspect cgit is timing out because it's looking for enough logs to fill ! a single page before sending a reply. That is probably timing out after some fixed time, maybe 60 sec. ! This is an advantage of github's ! progressive rendering (and the pile of javascript it requires). For ! example, see: ! ! https://github.com/freebsd/freebsd-ports/commits/main/devel/py-python-dateutil ! ! where you get the first commit and then a spinning thing for a bit ! before "End of commit history for this file" appears. That is different, indeed. I suppose different preprocessing: whatever git needs to compute during these 80 seconds, is already cached there. The javascript (presentation layer) is a different matter - what I am asking is, what is computed within the 80 seconds and how does that happen? Because I have not seen similar before. Maybe it has a simple explanation, maybe it will be optimized away at some point, or maybe it gets worse...