From nobody Mon Apr 08 04:06:30 2024 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VCbBG3v3Hz5G3Y4; Mon, 8 Apr 2024 04:06:30 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VCbBG1lyWz4Zk3; Mon, 8 Apr 2024 04:06:30 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1712549190; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=7l+lmMmPORTWAi+QtBrz+fIuApnDfbw1kcXg1Gup7Bc=; b=kuN86Puq77A0D3/+LgmJf3W0hm0atyFuhoknH4Rm7+L5KCj7JWDy2qGasru127fTiDYrpn 01pQE5vZGIs82cRk2Sl65TwXtdAnpZYzISbzj6ks5ubxif20ZcaWjf1Ymy1gnG74ahEQf/ nnnApUxjvKynT6fbzzuDJJ63/VR1Epj3S25axWAwIieucG1qbxzKd1N7FgvX0f+SohIono 1DatinrouMmzZdwP6oLcnp4BV3RMSDrLK8g0BAf90jsRpY9iyxtPrWnnf4+s+1BhlXVe/v pzktrCZCeRjrsCudWGAwE5G2IIEFESmkzhZnkujdIUK8mP0+4O3DD6BKtgvs3A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1712549190; a=rsa-sha256; cv=none; b=DE7mlQ9Z5tjKWaB2qSMI/fbWjzqHpkzO0JOIOXYoYHuU/HCW+2Lxq8YwMQF+WUi+xaRWmd BlMENI3jDE1i4qK33gm+69ovMZoXnIAOv9VznnUNUdsMmxtI1zwX8PnigIbWKLhhLJqHs6 hvBGxIOGY83vxPiz+lRQ2I7Qahxga3xA3wdWI2hR5v4Vy4XAQcxm9NZVQmAL8RagedP89d RZp2RSzHwcthi+r3/vhF+B8nzRH0iwn8Prmy+kOmSg7REst4HETfAXIYIG3xlR/S1tqlsE Cd94sjXIdCX6uA5dtOlMocKU0UbxHq6n5PT0LZc+hzGwfd+6YK5opd5vX/kDQQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1712549190; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=7l+lmMmPORTWAi+QtBrz+fIuApnDfbw1kcXg1Gup7Bc=; b=ejzn71rNFXF+g/k9JgPooVJgsUOsJLsN3PHCaCSd+nA+i7jv2oavsMxKsn8kAAerM664GG 3n1X1Vg1m4mIQems+pSDf9uT2PA1xEsJGv1xUalFVmZQ8Ereim8Ydo8rOzizPxxNT6p9Ov 3n0RkC9ib5uGmJ8ncFY4mI7q+Ym/Dj24CIMyIIv4OIeJy5VnrlT+UWs0KBZrT36MVN7e5k ClTlPytee/cfxMJTQraoHuNz6l0v74nWFbNvFHfoR4lZssidXycD8143xWu4NE9nZGxB1C 9F6MxiTGuEJZvYqbTl6dBCtgE+/7TrzQm0FE1LO8GDdlb+GGQ4/y/uDZfZx7Ow== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4VCbBG1BHhzwj6; Mon, 8 Apr 2024 04:06:30 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 43846Ui4062386; Mon, 8 Apr 2024 04:06:30 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 43846U7J062383; Mon, 8 Apr 2024 04:06:30 GMT (envelope-from git) Date: Mon, 8 Apr 2024 04:06:30 GMT Message-Id: <202404080406.43846U7J062383@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Xin LI Subject: git: 728293e74c90 - stable/13 - Backport export of lzma_mt_block_size symbol. List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: delphij X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 728293e74c90d45d7901dfd40c4872d7e6a787eb Auto-Submitted: auto-generated The branch stable/13 has been updated by delphij: URL: https://cgit.FreeBSD.org/src/commit/?id=728293e74c90d45d7901dfd40c4872d7e6a787eb commit 728293e74c90d45d7901dfd40c4872d7e6a787eb Author: Xin LI AuthorDate: 2024-04-05 06:39:33 +0000 Commit: Xin LI CommitDate: 2024-04-08 04:06:13 +0000 Backport export of lzma_mt_block_size symbol. This restores binary compatibility against liblzma 5.6.0 library. PR: 278127 (cherry picked from commit 5ffb19ac36c4b5c9e72d69048250c8e56acff739) --- contrib/xz/src/liblzma/api/lzma/container.h | 28 +++++++++++++++++++++++ contrib/xz/src/liblzma/common/filter_encoder.c | 16 ++++++++----- contrib/xz/src/liblzma/common/filter_encoder.h | 6 +---- contrib/xz/src/liblzma/common/stream_encoder_mt.c | 20 ++++++++-------- contrib/xz/src/liblzma/liblzma_generic.map | 5 ++++ contrib/xz/src/liblzma/liblzma_linux.map | 5 ++++ contrib/xz/src/liblzma/lz/lz_encoder.c | 4 +--- contrib/xz/src/liblzma/lz/lz_encoder.h | 8 +++++++ contrib/xz/src/liblzma/lzma/lzma2_encoder.c | 3 +++ lib/liblzma/Symbol.map | 5 +++- lib/liblzma/Versions.def | 5 +++- 11 files changed, 78 insertions(+), 27 deletions(-) diff --git a/contrib/xz/src/liblzma/api/lzma/container.h b/contrib/xz/src/liblzma/api/lzma/container.h index 2849fbfd3c51..4cbb1b7bf7b4 100644 --- a/contrib/xz/src/liblzma/api/lzma/container.h +++ b/contrib/xz/src/liblzma/api/lzma/container.h @@ -435,6 +435,34 @@ extern LZMA_API(lzma_ret) lzma_stream_encoder_mt( lzma_nothrow lzma_attr_warn_unused_result; +/** + * \brief Calculate recommended Block size for multithreaded .xz encoder + * + * This calculates a recommended Block size for multithreaded encoding given + * a filter chain. This is used internally by lzma_stream_encoder_mt() to + * determine the Block size if the block_size member is not set to the + * special value of 0 in the lzma_mt options struct. + * + * If one wishes to change the filters between Blocks, this function is + * helpful to set the block_size member of the lzma_mt struct before calling + * lzma_stream_encoder_mt(). Since the block_size member represents the + * maximum possible Block size for the multithreaded .xz encoder, one can + * use this function to find the maximum recommended Block size based on + * all planned filter chains. Otherwise, the multithreaded encoder will + * base its maximum Block size on the first filter chain used (if the + * block_size member is not set), which may unnecessarily limit the Block + * size for a later filter chain. + * + * \param filters Array of filters terminated with + * .id == LZMA_VLI_UNKNOWN. + * + * \return Recommended Block size in bytes, or UINT64_MAX if + * an error occurred. + */ +extern LZMA_API(uint64_t) lzma_mt_block_size(const lzma_filter *filters) + lzma_nothrow; + + /** * \brief Initialize .lzma encoder (legacy file format) * diff --git a/contrib/xz/src/liblzma/common/filter_encoder.c b/contrib/xz/src/liblzma/common/filter_encoder.c index 46fe8af1c153..0699bcee62a4 100644 --- a/contrib/xz/src/liblzma/common/filter_encoder.c +++ b/contrib/xz/src/liblzma/common/filter_encoder.c @@ -33,7 +33,8 @@ typedef struct { /// Calculates the recommended Uncompressed Size for .xz Blocks to /// which the input data can be split to make multithreaded /// encoding possible. If this is NULL, it is assumed that - /// the encoder is fast enough with single thread. + /// the encoder is fast enough with single thread. If the options + /// are invalid, UINT64_MAX is returned. uint64_t (*block_size)(const void *options); /// Tells the size of the Filter Properties field. If options are @@ -248,26 +249,29 @@ lzma_raw_encoder_memusage(const lzma_filter *filters) } -extern uint64_t +extern LZMA_API(uint64_t) lzma_mt_block_size(const lzma_filter *filters) { + if (filters == NULL) + return UINT64_MAX; + uint64_t max = 0; for (size_t i = 0; filters[i].id != LZMA_VLI_UNKNOWN; ++i) { const lzma_filter_encoder *const fe = encoder_find(filters[i].id); + if (fe == NULL) + return UINT64_MAX; + if (fe->block_size != NULL) { const uint64_t size = fe->block_size(filters[i].options); - if (size == 0) - return 0; - if (size > max) max = size; } } - return max; + return max == 0 ? UINT64_MAX : max; } diff --git a/contrib/xz/src/liblzma/common/filter_encoder.h b/contrib/xz/src/liblzma/common/filter_encoder.h index f1d5683fe793..da92be8b34da 100644 --- a/contrib/xz/src/liblzma/common/filter_encoder.h +++ b/contrib/xz/src/liblzma/common/filter_encoder.h @@ -1,6 +1,6 @@ /////////////////////////////////////////////////////////////////////////////// // -/// \file filter_encoder.c +/// \file filter_encoder.h /// \brief Filter ID mapping to filter-specific functions // // Author: Lasse Collin @@ -16,10 +16,6 @@ #include "common.h" -// FIXME: Might become a part of the public API. -extern uint64_t lzma_mt_block_size(const lzma_filter *filters); - - extern lzma_ret lzma_raw_encoder_init( lzma_next_coder *next, const lzma_allocator *allocator, const lzma_filter *filters); diff --git a/contrib/xz/src/liblzma/common/stream_encoder_mt.c b/contrib/xz/src/liblzma/common/stream_encoder_mt.c index f64de9bdbc57..64de526b6a17 100644 --- a/contrib/xz/src/liblzma/common/stream_encoder_mt.c +++ b/contrib/xz/src/liblzma/common/stream_encoder_mt.c @@ -979,20 +979,18 @@ get_options(const lzma_mt *options, lzma_options_easy *opt_easy, *filters = opt_easy->filters; } - // Block size - if (options->block_size > 0) { - if (options->block_size > BLOCK_SIZE_MAX) - return LZMA_OPTIONS_ERROR; - + // If the Block size is not set, determine it from the filter chain. + if (options->block_size > 0) *block_size = options->block_size; - } else { - // Determine the Block size from the filter chain. + else *block_size = lzma_mt_block_size(*filters); - if (*block_size == 0) - return LZMA_OPTIONS_ERROR; - assert(*block_size <= BLOCK_SIZE_MAX); - } + // UINT64_MAX > BLOCK_SIZE_MAX, so the second condition + // should be optimized out by any reasonable compiler. + // The second condition should be there in the unlikely event that + // the macros change and UINT64_MAX < BLOCK_SIZE_MAX. + if (*block_size > BLOCK_SIZE_MAX || *block_size == UINT64_MAX) + return LZMA_OPTIONS_ERROR; // Calculate the maximum amount output that a single output buffer // may need to hold. This is the same as the maximum total size of diff --git a/contrib/xz/src/liblzma/liblzma_generic.map b/contrib/xz/src/liblzma/liblzma_generic.map index bb82167ed57a..b251d366e15c 100644 --- a/contrib/xz/src/liblzma/liblzma_generic.map +++ b/contrib/xz/src/liblzma/liblzma_generic.map @@ -119,3 +119,8 @@ global: lzma_str_list_filters; lzma_str_to_filters; } XZ_5.2; + +XZ_5.5.0alpha { +global: + lzma_mt_block_size; +} XZ_5.4; diff --git a/contrib/xz/src/liblzma/liblzma_linux.map b/contrib/xz/src/liblzma/liblzma_linux.map index 449f5fd682db..25b393883693 100644 --- a/contrib/xz/src/liblzma/liblzma_linux.map +++ b/contrib/xz/src/liblzma/liblzma_linux.map @@ -134,3 +134,8 @@ global: lzma_str_list_filters; lzma_str_to_filters; } XZ_5.2; + +XZ_5.5.0alpha { +global: + lzma_mt_block_size; +} XZ_5.4; diff --git a/contrib/xz/src/liblzma/lz/lz_encoder.c b/contrib/xz/src/liblzma/lz/lz_encoder.c index 5489085a0860..8e724a035a13 100644 --- a/contrib/xz/src/liblzma/lz/lz_encoder.c +++ b/contrib/xz/src/liblzma/lz/lz_encoder.c @@ -196,9 +196,7 @@ lz_encoder_prepare(lzma_mf *mf, const lzma_allocator *allocator, // For now, the dictionary size is limited to 1.5 GiB. This may grow // in the future if needed, but it needs a little more work than just // changing this check. - if (lz_options->dict_size < LZMA_DICT_SIZE_MIN - || lz_options->dict_size - > (UINT32_C(1) << 30) + (UINT32_C(1) << 29) + if (!IS_ENC_DICT_SIZE_VALID(lz_options->dict_size) || lz_options->nice_len > lz_options->match_len_max) return true; diff --git a/contrib/xz/src/liblzma/lz/lz_encoder.h b/contrib/xz/src/liblzma/lz/lz_encoder.h index ffcba02ce931..b71f11805e50 100644 --- a/contrib/xz/src/liblzma/lz/lz_encoder.h +++ b/contrib/xz/src/liblzma/lz/lz_encoder.h @@ -17,6 +17,14 @@ #include "common.h" +// For now, the dictionary size is limited to 1.5 GiB. This may grow +// in the future if needed, but it needs a little more work than just +// changing this check. +#define IS_ENC_DICT_SIZE_VALID(size) \ + ((size) >= LZMA_DICT_SIZE_MIN \ + && (size) <= (UINT32_C(1) << 30) + (UINT32_C(1) << 29)) + + /// A table of these is used by the LZ-based encoder to hold /// the length-distance pairs found by the match finder. typedef struct { diff --git a/contrib/xz/src/liblzma/lzma/lzma2_encoder.c b/contrib/xz/src/liblzma/lzma/lzma2_encoder.c index 4b6b23118d70..5043a07e0fdf 100644 --- a/contrib/xz/src/liblzma/lzma/lzma2_encoder.c +++ b/contrib/xz/src/liblzma/lzma/lzma2_encoder.c @@ -409,6 +409,9 @@ lzma_lzma2_block_size(const void *options) { const lzma_options_lzma *const opt = options; + if (!IS_ENC_DICT_SIZE_VALID(opt->dict_size)) + return UINT64_MAX; + // Use at least 1 MiB to keep compression ratio better. return my_max((uint64_t)(opt->dict_size) * 3, UINT64_C(1) << 20); } diff --git a/lib/liblzma/Symbol.map b/lib/liblzma/Symbol.map index 9532da0a5bde..938b6191b4f4 100644 --- a/lib/liblzma/Symbol.map +++ b/lib/liblzma/Symbol.map @@ -114,6 +114,10 @@ XZ_5.4 { lzma_str_to_filters; }; +XZ_5.6 { + lzma_mt_block_size; +}; + XZprivate_1.0 { lzma_alloc; lzma_alloc_zero; @@ -173,7 +177,6 @@ XZprivate_1.0 { lzma_mf_hc3_skip; lzma_mf_hc4_find; lzma_mf_hc4_skip; - lzma_mt_block_size; lzma_next_end; lzma_next_filter_init; lzma_next_filter_update; diff --git a/lib/liblzma/Versions.def b/lib/liblzma/Versions.def index dea89514beda..25064d9d50e7 100644 --- a/lib/liblzma/Versions.def +++ b/lib/liblzma/Versions.def @@ -8,6 +8,9 @@ XZ_5.2 { XZ_5.4 { } XZ_5.2; -XZprivate_1.0 { +XZ_5.6 { } XZ_5.4; +XZprivate_1.0 { +} XZ_5.6; +