git: ec2764d5ec49 - main - Mk/bsd.port.mk: prepare the land for pkg 1.17

[Kevin Bowling]

On Thu, Jul 22, 2021 at 1:46 AM Stefan Esser <se at freebsd.org> wrote:

> Am 22.07.21 um 04:35 schrieb Matthew D. Fuller:
> > On Thu, Jul 22, 2021 at 02:12:08AM +0000 I heard the voice of
> > Alexey Dokuchaev, and lo! it spake thus:
> >>
> >> I've switched to PKG_NOCOMPRESS and
> >> PKG_DEPENDS+=lbzip2:archivers/lbzip2 since 2017 when I got
> >> dissatisfied with ridiculously long LLVM packaging times*
> >
> > I've wished (not enough yet to see if I could manage it, but) to be
> > able to use something like `gzip -1` for compression on some poudriere
> > instances; all the clients pulling from it are on the same LAN if not
> > the same physical machine, so bandwidth is trivially enough, and
> > compression/decompression times eat up a significant amount of the
> > cost of updates...
>
> I have used .tar without compression on ZFS file systems that use zstd-2
> compression (my default for all file systems not meant to hold already
> compressed distfiles or media files).
>
> This is the fastest and most efficient configuration IMHO (if you are
> using ZFS anyway), with an effective compression ratio of more than 3
> in practice and the compression time hidden in the filesystem layer.
>

I’m skeptical of the performance of the zstd implementation in openzfs as
it stands, you may lose out on overall system performance by using it right
now.  I believe mjg@ did some tests with buildworld and poudriere and found
the pool threading design in it doesn’t scale up well (see
https://github.com/openzfs/zfs/blob/master/module/zstd/zfs_zstd.c).  So a
user of poudriere would probably not want to do it like this, if you can
build up parallelism userspace is the right place to do the compression.


> If you are transferring these files over slow networks, that is not a
> good setup, but within a system or on a LAN it does not matter.
>
> Example of a randomly selected old package file of significant size:
>
> $ cd /usr/ports/portmaster-backups
> $ time cp llvm60-6.0.1_1.tar /tmp/
>
> real    0m0.928s
> user    0m0.000s
> sys     0m0.930s
>
> $ time cp /tmp/llvm60-6.0.1_1.tar .
>
> real    0m0.572s
> user    0m0.000s
> sys     0m0.575s
>
> $ ls -ls llvm60-6.0.1_1.tar /tmp/llvm60-6.0.1_1.tar
> 835724 -rw-r--r--  1 root  wheel  855781376 Jul 22 09:57
> /tmp/llvm60-6.0.1_1.tar
> 290613 -rw-r--r--  1 root  wheel  855781376 Jul 22 09:58 llvm60-6.0.1_1.tar
>
> The compression ratio is about 2.9, and the system time required for
> reading the file (the copy to /tmp on tmpfs) is about doubled due
> to the decompression performed by ZFS (compared to copying from a
> ZFS file system without compression).
>
> The compression time is not (fully) accounted to the process writing
> the file to the compressed file system, but the real time is what
> you observe from the point of view of the build process.
>
> For comparison with user land compression
>
> $ cd /tmp
> $ time bzip2 -1 < llvm60-6.0.1_1.tar > llvm60-6.0.1_1.tbz
>
> real    0m57.487s
> user    0m57.318s
> sys     0m0.169s
>
> $ ls -ls llvm60-6.0.1_1.tbz
> 241348 -rw-r--r--  1 root  wheel  247136911 Jul 22 10:15 llvm60-6.0.1_1.tbz
>
> $ time xz -1 < llvm60-6.0.1_1.tar > llvm60-6.0.1_1.txz
>
> real    0m28.899s
> user    0m28.714s
> sys     0m0.183s
>
> $ ls -ls llvm60-6.0.1_1.txz
> 191412 -rw-r--r--  1 root  wheel  196004988 Jul 22 10:17 llvm60-6.0.1_1.txz
>
> $ time xz -9 < llvm60-6.0.1_1.tar > llvm60-6.0.1_1.txz
>
> real    3m41.291s
> user    3m40.973s
> sys     0m0.314s
>
> $ ls -ls /tmp/llvm60-6.0.1_1.txz
> 104196 -rw-r--r--  1 root  wheel  106694296 Jul 22 10:25 llvm60-6.0.1_1.txz
>
> The compression achieved by using XZ is much higher than that of zstd-2,
> but the time saving of using tar instead of txz is substantial, too.
>
> I have just performed a test with zstd-19 for /usr/packages and found, that
> the compression ratio is within 33% of that of xz -1 at much higher speed
> (and nearly as good as bzip -1), and I'm considering to keep this setting
> for /usr/packages:
>
> $ ls -ls llvm60-6.0.1_1.tar
> 254307 -rw-r--r--  1 root  wheel  855781376 Jul 22 10:28 llvm60-6.0.1_1.tar
>
> Regards, STefan
>
>