From nobody Fri Nov 26 10:34:46 2021 X-Original-To: dev-commits-ports-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 571CB18AE7B8; Fri, 26 Nov 2021 10:34:47 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4J0rj271ffz4RNq; Fri, 26 Nov 2021 10:34:46 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id D19F32F13F; Fri, 26 Nov 2021 10:34:46 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 1AQAYk4D095149; Fri, 26 Nov 2021 10:34:46 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 1AQAYkZ1095148; Fri, 26 Nov 2021 10:34:46 GMT (envelope-from git) Date: Fri, 26 Nov 2021 10:34:46 GMT Message-Id: <202111261034.1AQAYkZ1095148@gitrepo.freebsd.org> To: ports-committers@FreeBSD.org, dev-commits-ports-all@FreeBSD.org, dev-commits-ports-main@FreeBSD.org From: Tobias Kortkamp Subject: git: b6633a0113fd - main - ports-mgmt/parfetch-overlay: Add overlay to fetch distfiles in parallel and with pipelining List-Id: Commits to the main branch of the FreeBSD ports repository List-Archive: https://lists.freebsd.org/archives/dev-commits-ports-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-ports-main@freebsd.org X-BeenThere: dev-commits-ports-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: tobik X-Git-Repository: ports X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: b6633a0113fd4d12b1310a02a865dc1e6c5f0169 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1637922887; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=c1/CTEsnkrZsX/2b8+TU2XrxP7IwPHM3oNTL6O5nhVE=; b=nG5EUjdP4OmC22xGsQ3jvf47SbN4s/YML9eLGOY3F3IV+CoXzrlEjoHAgFKtR65c6sI5y5 hHstLyAzAilAcCwl1Y1XKiinkYt4Df+TXh/eZTWXes9WELFvo6zCAbNTY+ztqRZR061hI2 w4m68fM1M/QQm/+JaogZ64vjKSIt5q3eYunrB6fmGORbMrdJ040xnwCs6tFj6UItAY5J+/ kIRqBNqVvRyxZeO/F1U0FZVnlJ5xo4AeFSc5lFkd6gl+m+5VgE6tjZ4XCLwii6qAZIsdOA ncp5xvdCF14WcM1pNC106L3Dejzeq85HF6Gr7qReBLkE3Px0x67wdpbDQoNGuw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1637922887; a=rsa-sha256; cv=none; b=dqtEU3S23eV4aEtJPf2Ab2i3k77uaB8de50btokysFYoOm9jbv1ITnRuhFVn3oOvwnlGvx yPR/gg2b4sq3MFndGDlqcXrBtoHfHVrVzY8BAU43bExLBIcH5OT5bdHLNxELQAdek8CM4a +ra5TV9WKRSKiuDthfh4aVP4Ry54RAE+wL5F6MDjWMyp+nmj8C8SqDj7qBvSaXIgLO1TMb KP971AhhKarSKntS8UVfXWCKS/tNlVUInBgQfdDkuYEXKMhwm5cB3s17vttedZic9pV9dX dOVonJt9FosKTmEC9Euox2xeIsUgmLlaSfA8oJN2ktBxqbjbRys+vTlVEsP7FQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by tobik: URL: https://cgit.FreeBSD.org/ports/commit/?id=b6633a0113fd4d12b1310a02a865dc1e6c5f0169 commit b6633a0113fd4d12b1310a02a865dc1e6c5f0169 Author: Tobias Kortkamp AuthorDate: 2021-11-26 10:21:50 +0000 Commit: Tobias Kortkamp CommitDate: 2021-11-26 10:33:16 +0000 ports-mgmt/parfetch-overlay: Add overlay to fetch distfiles in parallel and with pipelining Ports that fetch multiple distfiles from the same host can especially benefit from this. This is a glue application between libcurl and the ports framework. It comes with a ports overlay that replaces the default do-fetch and checksum targets. This is an experiment. Use at your own risk. WWW: https://github.com/t6/parfetch Demo: https://asciinema.org/a/ogjs0MiCNZxqwYy2xliHfYEaM Some unscientific but repeatable numbers: tokei (152 distfiles) fetch is down from ~44s to ~3s. wlroots (34 distfiles) fetch is down from ~12s to ~3s. rust makesum (28 distfiles) is down from ~90s to ~50s. --- ports-mgmt/Makefile | 1 + ports-mgmt/parfetch-overlay/Makefile | 29 +++++++++++++++++++ ports-mgmt/parfetch-overlay/distinfo | 3 ++ ports-mgmt/parfetch-overlay/files/pkg-message.in | 37 ++++++++++++++++++++++++ ports-mgmt/parfetch-overlay/pkg-descr | 12 ++++++++ ports-mgmt/parfetch-overlay/pkg-plist | 4 +++ 6 files changed, 86 insertions(+) diff --git a/ports-mgmt/Makefile b/ports-mgmt/Makefile index 3809ff9946d0..fed51cb0637b 100644 --- a/ports-mgmt/Makefile +++ b/ports-mgmt/Makefile @@ -21,6 +21,7 @@ SUBDIR += p5-FreeBSD-Ports SUBDIR += p5-FreeBSD-Ports-INDEXhash SUBDIR += packagekit + SUBDIR += parfetch-overlay SUBDIR += pchecker SUBDIR += pkg SUBDIR += pkg-devel diff --git a/ports-mgmt/parfetch-overlay/Makefile b/ports-mgmt/parfetch-overlay/Makefile new file mode 100644 index 000000000000..a14ac93ca17a --- /dev/null +++ b/ports-mgmt/parfetch-overlay/Makefile @@ -0,0 +1,29 @@ +PORTNAME= parfetch +DISTVERSION= a20211126 +CATEGORIES= ports-mgmt +MASTER_SITES= https://github.com/t6/parfetch/releases/download/${DISTVERSION}/ +PKGNAMESUFFIX= -overlay + +MAINTAINER= tobik@FreeBSD.org +COMMENT= Fetch distfiles in parallel and with pipelining + +LICENSE= BSD2CLAUSE +LICENSE_FILE= ${WRKSRC}/LICENSE + +LIB_DEPENDS= libcurl.so:ftp/curl \ + libevent.so:devel/libevent + +USES= ninja pkgconfig tar:lz +HAS_CONFIGURE= yes +CONFIGURE_ARGS= AR="${AR}" \ + CC="${CC}" \ + CFLAGS="${CFLAGS}" \ + CPPFLAGS="${CPPFLAGS}" \ + DESTDIR="${STAGEDIR}" \ + INSTALL_DATA="${INSTALL_DATA}" \ + INSTALL_PROGRAM="${INSTALL_PROGRAM}" \ + LDFLAGS="${LDFLAGS}" \ + PREFIX="${PREFIX}" +SUB_FILES= pkg-message + +.include diff --git a/ports-mgmt/parfetch-overlay/distinfo b/ports-mgmt/parfetch-overlay/distinfo new file mode 100644 index 000000000000..bea39c4e4a8a --- /dev/null +++ b/ports-mgmt/parfetch-overlay/distinfo @@ -0,0 +1,3 @@ +TIMESTAMP = 1637920935 +SHA256 (parfetch-a20211126.tar.lz) = 55c3614b57aeb8664c11e6c7c8a0bffd5cace6fb3a131d2c504539e31dca4d90 +SIZE (parfetch-a20211126.tar.lz) = 83026 diff --git a/ports-mgmt/parfetch-overlay/files/pkg-message.in b/ports-mgmt/parfetch-overlay/files/pkg-message.in new file mode 100644 index 000000000000..896670f980e4 --- /dev/null +++ b/ports-mgmt/parfetch-overlay/files/pkg-message.in @@ -0,0 +1,37 @@ +[ +{ type: install + message: <