Re: /usr/ports/distfiles/go insanely oversized

Hello!

On 6/17/24 10:26, Kevin Oberman wrote:
> I'm baffled. My distfiles with over 1200 ports installed total 9.1GB. 
> This makes me wonder just why you are doing things like 'make 
> checksum'? 'make install' will do the following:
> config, fetch, checksum, depends, extract, patch, configure, build, 
> install.
"make fetch" and "make checksum" require supervision due to 
peculiarities of local Internet that I am afraid to discuss. After this, 
"make install" can be run unattended.
>
> That normally results in a single file in distfiles. Some ports 
> require multiple dist files, usually for a port which maintains 
> multiple versions or a metaport that contains distfiles for many ports 
> included in the metaport. E.g. gnome. Still, this does not mean that 
> there will be anything like what you are seeing.

It is exactly so for everything but NOT for /usr/ports/distfiles/go

I see that "make checksum" fetches and checksums not only the files 
listed in, say, /usr/ports/archivers/nfpm/distinfo but some (lots of) 
files placed in /usr/ports/distfiles/go/archivers_nfpm/pkg/mod/cache - 
and I cannot understand why.

>
> Can you explain exactly how you are doing this?

For instance (I use archivers/nfpm as a testbed for this problem - it's 
the first one in alphabetical order):

# cd /usr/ports/archivers/nfpm

# make fetch

# make checksum

# make install

> And, why?
I am afraid of discussing the "why". Hint: HTTP error 451
> Why not just use packages?
I do it often but not always. Sometimes I need ports.
> Either pull down the packages you need or use poudriere or 'make 
> package' to generate what you need? Either way will be far less 
> resource intensive. Especially pulling down the pre-built packages.
>
> Clearly, something you're doing is not what you want.
Yours, Anatoly
>
> On Sun, Jun 16, 2024 at 2:16 AM Thor Ablestar <thor@irk.ru> wrote:
>
>     Hello!
>
>     I have a habit to keep /usr/ports/distfiles on separate SSD in
>     order to
>     build packages on many different computers and OSes (12, 13 and now
>     14.2). Usually i do the following:
>
>     # cd /usr/ports/sometheme/somepackage
>
>     # make checksum
>
>     # make install
>
>     so that I supervise the download of source files and then have
>     some rest
>     while the port compiles.
>
>     SSD became full, and I decided to copy data to the different HDD.
>     Then I
>     have found that the /usr/ports/distfiles/go has an insane number of
>     subdirectories (some hundreds of thousands). I spent a full day in
>     attempts to copy them (successfully at last) and only some minutes to
>     copy the rest.
>
>     Then I decided to find a total size of the distfiles with du. Now du
>     spent 5 minutes of machine time (top listing) and has shown 288 GB
>     in go
>     subdir and 469 GB total.
>
>     Then I have tried to compare some subdirectories in go directory,
>     and it
>     seems that each subdirectory contains subdirectories of all their
>     dependencies (I am not sure). For instance,
>
>     go/archivers_nfpm/pkg/mod/cache/download/github.com/!masterminds/goutils/@v
>     <http://github.com/!masterminds/goutils/@v>
>
>     contains the same files (name and size) as
>
>     go/audio_gonic/pkg/mod/cache/download/github.com/!masterminds/goutils/@v
>     <http://github.com/!masterminds/goutils/@v>
>
>     What to do? How to manage it? If the problem cannot be solved I am
>     going
>     just to ban the /usr/ports/distfiles/go dependent ports completely
>     and
>     try to live without them.
>
>     I had the same problem 2 years ago but it was not so severe as now.
>
>     Sincerely yours,
>
>     Anatoly
>
>
>
>
>
> -- 
> Kevin Oberman, Part time kid herder and retired Network Engineer
> E-mail: rkoberman@gmail.com
> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683