From nobody Wed Jan 17 15:56:15 2024 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TFVq33PqWz5859k; Wed, 17 Jan 2024 15:56:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TFVq32XDTz4DKB; Wed, 17 Jan 2024 15:56:15 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1705506975; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=WO5DP8R3dHABc8BcMM0wtrqO38nUIvhmB5R5CUSzMR0=; b=gMAsPXK4H8/JbbtsV/bAHuA88wZbUQ00XeCpGIYKOO2mnfcSAXx4qqKZVlQebnOqVhK5+Y I8gsYivDc3s4pPCz93GI/jzAY+K4mUUOsxugIQe+dmLECGKlTcl8iUKFgZwmMrVYLxIvkJ QRar18rzqR8pYLjCT5e3Oqmg3vUNS3g6X2ptM/AG00MaaTuARxTjqsp41bMStX7jFmj/9g AvFNu5OiOUKdsO6FqmKD/6E31RnfBq2Y/kK9SWdmIBwXYLuirDtqkMQQKt+7F1AR/vy0ea 5izLS3oyF/6VNecNWR+ZSVN0Evltlo+YOxqzQRZJIftqp0Z6ZoUsp6fg0AbHvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1705506975; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=WO5DP8R3dHABc8BcMM0wtrqO38nUIvhmB5R5CUSzMR0=; b=jq5x2Vqpi5789XDLQp81pIuO0S2J06u182/dH/ZcrZs8T+89GsZGuvSdtSzbyDgCLQl+Vp o2XrxX5f7fWWxlWaXZiTWSBDnHKWT5uzFjqBZ3yb6VohLdus3k/FGbq0TRGdoWK1PTa5Lo 0skMtfOWJnPDJDvKN/80EnAXZkLIaxCFU4Lfbhl9e8U4rlB3QL0qDQ3HNccVDZBA58qPI/ BlUiAB1rnpGK9qpmnscPLoyGF7/mgUSuEsr9iWXcMTxk6SexutUWqJzufj9qAebYr03gkT JrvKcTRGGgPyUWXxb3aC/qQoBlAcRSHHrIzC49S+d6KhJl9mZqPMRtbxey0wxg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1705506975; a=rsa-sha256; cv=none; b=Sk7xwsTLeiDiWK27z1mhBKEzaYelEWlm8W6e0I+NbLfzirMBn3qd09h56fgeIGExWtnDwB ZXceV23xeJzxfGHm0/VOCgSwSasydl2yxBXCBGXznPQ2LogEGg5T4rL4jgaIpwkwFi823u ua0ZOxlYOVY5ZXrGKHGQD8EUNM+OyeIDApk0BLygCsYBrEGXf+FP1XO8XEpbb/xRKxxcB4 +e0U5m5sSG/OkosC5KA5WVly524ZPsQaTfdiL/qNDoFDxhn4bOTvnK8gknmYY2vhQCLo+V 8W/hxyWXcDI3jg7A/z/MUWl/9jH0S2Qv3Re2VCLJ5D0AAsHPhCfnJ0Yk7ecdOQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TFVq31b3XzVSK; Wed, 17 Jan 2024 15:56:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 40HFuFjc009871; Wed, 17 Jan 2024 15:56:15 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 40HFuFFE009868; Wed, 17 Jan 2024 15:56:15 GMT (envelope-from git) Date: Wed, 17 Jan 2024 15:56:15 GMT Message-Id: <202401171556.40HFuFFE009868@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?= Subject: git: 127f4dd60bed - stable/13 - gzip: Add support for decompressing zstd files. List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: des X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 127f4dd60bed2af6ab6b99560e7cce2e5ac86772 Auto-Submitted: auto-generated The branch stable/13 has been updated by des: URL: https://cgit.FreeBSD.org/src/commit/?id=127f4dd60bed2af6ab6b99560e7cce2e5ac86772 commit 127f4dd60bed2af6ab6b99560e7cce2e5ac86772 Author: Dag-Erling Smørgrav AuthorDate: 2022-11-02 13:41:44 +0000 Commit: Dag-Erling Smørgrav CommitDate: 2024-01-17 15:52:31 +0000 gzip: Add support for decompressing zstd files. Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D37236 (cherry picked from commit f9349d427431eaa5b3bd7e9e7e87725a138facd1) --- usr.bin/gzip/Makefile | 4 +- usr.bin/gzip/Makefile.depend | 1 + usr.bin/gzip/gzip.1 | 4 +- usr.bin/gzip/gzip.c | 60 ++++++++++++++++++++++------- usr.bin/gzip/unzstd.c | 89 ++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 143 insertions(+), 15 deletions(-) diff --git a/usr.bin/gzip/Makefile b/usr.bin/gzip/Makefile index b9e6d84f436e..33fbdb85d78c 100644 --- a/usr.bin/gzip/Makefile +++ b/usr.bin/gzip/Makefile @@ -5,7 +5,7 @@ PROG= gzip MAN= gzip.1 gzexe.1 zdiff.1 zforce.1 zmore.1 znew.1 -LIBADD= z lzma +LIBADD= z lzma zstd .if ${MK_BZIP2_SUPPORT} != "no" LIBADD+= bz2 @@ -13,6 +13,8 @@ LIBADD+= bz2 CFLAGS+= -DNO_BZIP2_SUPPORT .endif +CFLAGS+= -I${SRCTOP}/sys/contrib/zstd/lib + SCRIPTS= gzexe zdiff zforce zmore znew MLINKS+= gzip.1 gunzip.1 \ diff --git a/usr.bin/gzip/Makefile.depend b/usr.bin/gzip/Makefile.depend index 413518057565..7c13917770d9 100644 --- a/usr.bin/gzip/Makefile.depend +++ b/usr.bin/gzip/Makefile.depend @@ -9,6 +9,7 @@ DIRDEPS = \ lib/liblzma \ lib/libthr \ lib/libz \ + lib/libzstd \ .include diff --git a/usr.bin/gzip/gzip.1 b/usr.bin/gzip/gzip.1 index 8677edf2751a..afb1147f3dfe 100644 --- a/usr.bin/gzip/gzip.1 +++ b/usr.bin/gzip/gzip.1 @@ -23,7 +23,7 @@ .\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. -.Dd January 7, 2019 +.Dd November 2, 2022 .Dt GZIP 1 .Os .Sh NAME @@ -108,6 +108,7 @@ is also capable of decompressing files compressed using .Xr compress 1 , .Xr bzip2 1 , .Ar lzip , +.Xr zstd 1 , or .Xr xz 1 . .Sh OPTIONS @@ -201,6 +202,7 @@ Report progress to standard error. .Sh SEE ALSO .Xr bzip2 1 , .Xr compress 1 , +.Xr zstd 1 , .Xr xz 1 , .Xr fts 3 , .Xr zlib 3 diff --git a/usr.bin/gzip/gzip.c b/usr.bin/gzip/gzip.c index 873d571f2b9b..8eddac4adcdd 100644 --- a/usr.bin/gzip/gzip.c +++ b/usr.bin/gzip/gzip.c @@ -85,6 +85,9 @@ enum filetype { #endif #ifndef NO_LZ_SUPPORT FT_LZ, +#endif +#ifndef NO_ZSTD_SUPPORT + FT_ZSTD, #endif FT_LAST, FT_UNKNOWN @@ -117,6 +120,12 @@ enum filetype { #define LZ_MAGIC "LZIP" #endif +#ifndef NO_ZSTD_SUPPORT +#include +#define ZSTD_SUFFIX ".zst" +#define ZSTD_MAGIC "\050\265\057\375" +#endif + #define GZ_SUFFIX ".gz" #define BUFLEN (64 * 1024) @@ -163,6 +172,9 @@ static suffixes_t suffixes[] = { #endif #ifndef NO_LZ_SUPPORT SUFFIX(LZ_SUFFIX, ""), +#endif +#ifndef NO_ZSTD_SUPPORT + SUFFIX(ZSTD_SUFFIX, ""), #endif SUFFIX(GZ_SUFFIX, ""), /* Overwritten by -S "" */ #undef SUFFIX @@ -220,7 +232,7 @@ static const char *infile; /* name of file coming in */ static void maybe_err(const char *fmt, ...) __printflike(1, 2) __dead2; #if !defined(NO_BZIP2_SUPPORT) || !defined(NO_PACK_SUPPORT) || \ - !defined(NO_XZ_SUPPORT) + !defined(NO_XZ_SUPPORT) || !defined(NO_ZSTD_SUPPORT) static void maybe_errx(const char *fmt, ...) __printflike(1, 2) __dead2; #endif static void maybe_warn(const char *fmt, ...) __printflike(1, 2); @@ -283,6 +295,10 @@ static off_t unxz_len(int); static off_t unlz(int, int, char *, size_t, off_t *); #endif +#ifndef NO_ZSTD_SUPPORT +static off_t unzstd(int, int, char *, size_t, off_t *); +#endif + static const struct option longopts[] = { { "stdout", no_argument, 0, 'c' }, { "to-stdout", no_argument, 0, 'c' }, @@ -466,7 +482,7 @@ maybe_err(const char *fmt, ...) } #if !defined(NO_BZIP2_SUPPORT) || !defined(NO_PACK_SUPPORT) || \ - !defined(NO_XZ_SUPPORT) + !defined(NO_XZ_SUPPORT) || !defined(NO_ZSTD_SUPPORT) /* ... without an errno. */ void maybe_errx(const char *fmt, ...) @@ -1097,33 +1113,32 @@ file_gettype(u_char *buf) if (buf[0] == GZIP_MAGIC0 && (buf[1] == GZIP_MAGIC1 || buf[1] == GZIP_OMAGIC1)) return FT_GZIP; - else #ifndef NO_BZIP2_SUPPORT - if (memcmp(buf, BZIP2_MAGIC, 3) == 0 && + else if (memcmp(buf, BZIP2_MAGIC, 3) == 0 && buf[3] >= '0' && buf[3] <= '9') return FT_BZIP2; - else #endif #ifndef NO_COMPRESS_SUPPORT - if (memcmp(buf, Z_MAGIC, 2) == 0) + else if (memcmp(buf, Z_MAGIC, 2) == 0) return FT_Z; - else #endif #ifndef NO_PACK_SUPPORT - if (memcmp(buf, PACK_MAGIC, 2) == 0) + else if (memcmp(buf, PACK_MAGIC, 2) == 0) return FT_PACK; - else #endif #ifndef NO_XZ_SUPPORT - if (memcmp(buf, XZ_MAGIC, 4) == 0) /* XXX: We only have 4 bytes */ + else if (memcmp(buf, XZ_MAGIC, 4) == 0) /* XXX: We only have 4 bytes */ return FT_XZ; - else #endif #ifndef NO_LZ_SUPPORT - if (memcmp(buf, LZ_MAGIC, 4) == 0) + else if (memcmp(buf, LZ_MAGIC, 4) == 0) return FT_LZ; - else #endif +#ifndef NO_ZSTD_SUPPORT + else if (memcmp(buf, ZSTD_MAGIC, 4) == 0) + return FT_ZSTD; +#endif + else return FT_UNKNOWN; } @@ -1585,6 +1600,16 @@ file_uncompress(char *file, char *outfile, size_t outsize) size = unlz(fd, zfd, NULL, 0, NULL); break; #endif + +#ifndef NO_ZSTD_SUPPORT + case FT_ZSTD: + if (lflag) { + maybe_warnx("no -l with zstd files"); + goto lose; + } + size = unzstd(fd, zfd, NULL, 0, NULL); + break; +#endif case FT_UNKNOWN: if (lflag) { maybe_warnx("no -l for unknown filetypes"); @@ -1812,6 +1837,12 @@ handle_stdin(void) usize = unlz(STDIN_FILENO, STDOUT_FILENO, (char *)fourbytes, sizeof fourbytes, &gsize); break; +#endif +#ifndef NO_ZSTD_SUPPORT + case FT_ZSTD: + usize = unzstd(STDIN_FILENO, STDOUT_FILENO, + (char *)fourbytes, sizeof fourbytes, &gsize); + break; #endif } @@ -2187,6 +2218,9 @@ display_version(void) #ifndef NO_LZ_SUPPORT #include "unlz.c" #endif +#ifndef NO_ZSTD_SUPPORT +#include "unzstd.c" +#endif static ssize_t read_retry(int fd, void *buf, size_t sz) diff --git a/usr.bin/gzip/unzstd.c b/usr.bin/gzip/unzstd.c new file mode 100644 index 000000000000..4536f3119ace --- /dev/null +++ b/usr.bin/gzip/unzstd.c @@ -0,0 +1,89 @@ +/*- + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright (c) 2022 Klara, Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +/* This file is #included by gzip.c */ + +static off_t +unzstd(int in, int out, char *pre, size_t prelen, off_t *bytes_in) +{ + static char *ibuf, *obuf; + ZSTD_inBuffer zib; + ZSTD_outBuffer zob; + ZSTD_DCtx *zds; + ssize_t res; + size_t zres; + size_t bytes_out = 0; + int eof = 0; + + if (ibuf == NULL) + ibuf = malloc(BUFLEN); + if (obuf == NULL) + obuf = malloc(BUFLEN); + if (ibuf == NULL || obuf == NULL) + maybe_err("malloc"); + + zds = ZSTD_createDStream(); + ZSTD_initDStream(zds); + + zib.src = pre; + zib.size = prelen; + zib.pos = 0; + if (bytes_in != NULL) + *bytes_in = prelen; + zob.dst = obuf; + zob.size = BUFLEN; + zob.pos = 0; + + while (!eof) { + if (zib.pos >= zib.size) { + res = read(in, ibuf, BUFLEN); + if (res < 0) + maybe_err("read"); + if (res == 0) + eof = 1; + infile_newdata(res); + zib.src = ibuf; + zib.size = res; + zib.pos = 0; + if (bytes_in != NULL) + *bytes_in += res; + } + zres = ZSTD_decompressStream(zds, &zob, &zib); + if (ZSTD_isError(zres)) { + maybe_errx("%s", ZSTD_getErrorName(zres)); + } + if (zob.pos > 0) { + res = write(out, obuf, zob.pos); + if (res < 0) + maybe_err("write"); + zob.pos = 0; + bytes_out += res; + } + } + ZSTD_freeDStream(zds); + return (bytes_out); +}