From nobody Mon Jun 17 02:26:36 2024 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4W2Yg25Zn3z5N3Dq for ; Mon, 17 Jun 2024 02:26:54 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-yb1-xb2f.google.com (mail-yb1-xb2f.google.com [IPv6:2607:f8b0:4864:20::b2f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4W2Yg23Yhpz4gtF for ; Mon, 17 Jun 2024 02:26:54 +0000 (UTC) (envelope-from kob6558@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yb1-xb2f.google.com with SMTP id 3f1490d57ef6-dfef552d425so4587635276.0 for ; Sun, 16 Jun 2024 19:26:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1718591213; x=1719196013; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=9Dc/zoky+73/HL1H/4h43zRRxbCgDjcvNvxZ8iKx30Y=; b=Bv0DswTVZkBkSeYha5m2hzMdm1hdHiscUQuIs2Q+olsEa6B9PN2ObmCJvsh07fMjJl PQ3/dGPMUCYuvQU+08zYpFI7g4u4NtZbTA8z6ox2vJDPAZ0+ZFSM85n9klJA/J2HTs80 Yu0f3FKXAAbS9GYfm6rIaKUDVtCFXaN0ekQcsisHdjSbUHTX+xaBC8sHWhuvwoqwNXTy DD3/yQKVeBDE2Wr/gPE6k19VHLiPeneKxG1dZaoRYWsLZ6nRMGMoc2WdRkeYUaUcpcNu GqoAOs9T3qLJANDyMrspj/y5bGFascKsNa59CEfw+2Qb9pjkZtj7POWHHh5lYu1xCEL6 lG0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718591213; x=1719196013; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=9Dc/zoky+73/HL1H/4h43zRRxbCgDjcvNvxZ8iKx30Y=; b=vzTiesViw+ubmzY6Ma/OqYGXflaIQhXV4lUwC8WX1deL/s6xQfNPR7DNjXDXvybRO0 Ny+1qRLaPjIZYIsh7MsalB3HM93wa6mkK6JlXBglMThVKAZJUd9xztAlQUGffMuWZHgc /bcbc3YSpLwcbg3HSq48x3QpiA7tr23/rtnx+qCV3KHJ1bqH7Vp82NgkAGOpk1Z00Zkw /przMDTx+LMMFw9idwYWZaEQaMgjbxf9S3SoZCjEfgCVmfmMuBk2IIRpUurX80O8C2Wb jly6y+mks+nwNouZ5hXSiMh/k/GX0w6/fCzhmImWGY/Vi5jaXcHBXvt7H2Uxfmj22inZ 2ITw== X-Gm-Message-State: AOJu0Yxvjv5peFHTk2Dp7NAGRUe05t7p+mGkQYLxH8ATViCu9vfxtzn5 efs9XxuFJ7Hp5mhwh9ktBAQiHWfAV9e+QIPD2hzePNWmzfgONjLpJCJZpxjGm2d7BXFADlnELbP 7jrtU1Ky0RJ3wFmTK6eEkPsJug+a1a8fOQ4A= X-Google-Smtp-Source: AGHT+IEb7YK1j7FgYwL2IktNJ5NUfU9QrY1hgPxKgF0caL29TpkhkME/i3S3/zHl+rYxGkiTP36y4DnB/aOFzl5olWc= X-Received: by 2002:a25:81c6:0:b0:dfa:49dd:c55e with SMTP id 3f1490d57ef6-dff394e9356mr2665881276.62.1718591212943; Sun, 16 Jun 2024 19:26:52 -0700 (PDT) List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-questions@freebsd.org Sender: owner-freebsd-questions@FreeBSD.org MIME-Version: 1.0 References: <22e86a26-e2d5-47b3-8a28-26fde5c4d730@irk.ru> In-Reply-To: <22e86a26-e2d5-47b3-8a28-26fde5c4d730@irk.ru> From: Kevin Oberman Date: Sun, 16 Jun 2024 19:26:36 -0700 Message-ID: Subject: Re: /usr/ports/distfiles/go insanely oversized To: Thor Ablestar Cc: questions Content-Type: multipart/alternative; boundary="000000000000c300cf061b0cb2bf" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4W2Yg23Yhpz4gtF --000000000000c300cf061b0cb2bf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I'm baffled. My distfiles with over 1200 ports installed total 9.1GB. This makes me wonder just why you are doing things like 'make checksum'? 'make install' will do the following: config, fetch, checksum, depends, extract, patch, configure, build, install= . That normally results in a single file in distfiles. Some ports require multiple dist files, usually for a port which maintains multiple versions or a metaport that contains distfiles for many ports included in the metaport. E.g. gnome. Still, this does not mean that there will be anything like what you are seeing. Can you explain exactly how you are doing this? And, why? Why not just use packages? Either pull down the packages you need or use poudriere or 'make package' to generate what you need? Either way will be far less resource intensive. Especially pulling down the pre-built packages. Clearly, something you're doing is not what you want. On Sun, Jun 16, 2024 at 2:16=E2=80=AFAM Thor Ablestar wrote: > Hello! > > I have a habit to keep /usr/ports/distfiles on separate SSD in order to > build packages on many different computers and OSes (12, 13 and now > 14.2). Usually i do the following: > > # cd /usr/ports/sometheme/somepackage > > # make checksum > > # make install > > so that I supervise the download of source files and then have some rest > while the port compiles. > > SSD became full, and I decided to copy data to the different HDD. Then I > have found that the /usr/ports/distfiles/go has an insane number of > subdirectories (some hundreds of thousands). I spent a full day in > attempts to copy them (successfully at last) and only some minutes to > copy the rest. > > Then I decided to find a total size of the distfiles with du. Now du > spent 5 minutes of machine time (top listing) and has shown 288 GB in go > subdir and 469 GB total. > > Then I have tried to compare some subdirectories in go directory, and it > seems that each subdirectory contains subdirectories of all their > dependencies (I am not sure). For instance, > > go/archivers_nfpm/pkg/mod/cache/download/ > github.com/!masterminds/goutils/@v > > contains the same files (name and size) as > > go/audio_gonic/pkg/mod/cache/download/github.com/!masterminds/goutils/@v > > What to do? How to manage it? If the problem cannot be solved I am going > just to ban the /usr/ports/distfiles/go dependent ports completely and > try to live without them. > > I had the same problem 2 years ago but it was not so severe as now. > > Sincerely yours, > > Anatoly > > > > --=20 Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 --000000000000c300cf061b0cb2bf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I'm baffled. My distfiles with over 1200 por= ts installed total 9.1GB. This makes me wonder just why you are doing thing= s like 'make checksum'? 'make install' will do the followin= g:
config, fetch, checksum, depends, extract, patch, configu= re, build, install.

That normally re= sults in a single file in distfiles. Some ports require multiple dist files= , usually for a port which maintains multiple versions or a metaport that c= ontains distfiles for many ports included in the metaport. E.g. gnome. Stil= l, this does not mean that there will be anything like what you are seeing.=

Can you explain exactly how you are doi= ng this? And, why? Why not just use packages? Either pull down the packages= you need or use poudriere or 'make package' to generate what you n= eed? Either way will be far less resource intensive. Especially pulling dow= n the pre-built packages.

Clearly, somet= hing you're doing is not what you want.

On Sun, Jun 16, 2024= at 2:16=E2=80=AFAM Thor Ablestar <thor@i= rk.ru> wrote:
Hello!

I have a habit to keep /usr/ports/distfiles on separate SSD in order to build packages on many different computers and OSes (12, 13 and now
14.2). Usually i do the following:

# cd /usr/ports/sometheme/somepackage

# make checksum

# make install

so that I supervise the download of source files and then have some rest while the port compiles.

SSD became full, and I decided to copy data to the different HDD. Then I have found that the /usr/ports/distfiles/go has an insane number of
subdirectories (some hundreds of thousands). I spent a full day in
attempts to copy them (successfully at last) and only some minutes to
copy the rest.

Then I decided to find a total size of the distfiles with du. Now du
spent 5 minutes of machine time (top listing) and has shown 288 GB in go subdir and 469 GB total.

Then I have tried to compare some subdirectories in go directory, and it seems that each subdirectory contains subdirectories of all their
dependencies (I am not sure). For instance,

go/archivers_nfpm/pkg/mod/cache/download/github.com/!master= minds/goutils/@v

contains the same files (name and size) as

go/audio_gonic/pkg/mod/cache/download/github.com/!mastermin= ds/goutils/@v

What to do? How to manage it? If the problem cannot be solved I am going just to ban the /usr/ports/distfiles/go dependent ports completely and
try to live without them.

I had the same problem 2 years ago but it was not so severe as now.

Sincerely yours,

Anatoly





--
Kevin= Oberman, Part time kid herder and retired Network Engineer
E-mail: rkoberman@gmail.com
--000000000000c300cf061b0cb2bf--