From nobody Fri Nov 15 20:10:40 2024 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Xqp6n206Tz5dWdf; Fri, 15 Nov 2024 20:10:41 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Xqp6n19gvz4dHL; Fri, 15 Nov 2024 20:10:41 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731701441; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=r+xjo5tw2nSX4+eUczI/x9nEseBUD+aZhNaFp91BgnE=; b=ESRUU3yjk/+0/FAkEMd5H4q/7hR/yK3Rufg0ce3q6+Pn/F/lkEgZA2TSlHai7+8T88Dr0v VTBp/ctCNOWQB53FJI6A0kydrC6wH2zYaaI/pB9/XAfbPAN3Zcd5sR/82F+Hsb4EU+5ecD hB7M2LN/VRTC//jxXbcRgp+joOMgMVyD7w1b3KFLwQYoSZKchRExVzwtksg32mhZWKGEJe ew9ZgpxosorDsTcQD5tlZXkR+SkpSvNTTG8Hhz52NEci0vjRDL7LgUSNrkEi67wWWyEYTD dI2IGwJX9VmEwJCCPXBcdT/SrhePDQcAvg+57xzYGBrCFC/R5Rl6gBunJDcSCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731701441; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=r+xjo5tw2nSX4+eUczI/x9nEseBUD+aZhNaFp91BgnE=; b=wBK5feOO4LbUIh5z/urcXaAOCz8+UNcEB/Vb1o3QduEk6XGXZ3pC+E0vvjC0ZDLvFgGWol T8Qn2/lCSH8yMLcS2CwoPYBVh/v7WcVd7MZuFGj4qr0q/WWmKaQqFnVuLUSp7KER3i873h TAsItPkD41q8HN1uPN5Mh+mt4TZF7spynXT1j8bNl46V0Rj7eSIZM3EDEPpE3LPQNhbWiy O7RkR6d+1Pfc4ALqLAVj3CE/kyqwjaFVKON/2wKNe1T2Chd+elFF0h/HFPMehPsoRc11pE D2xsbZ7Do9QH3XlF9A0vf2hSwiSy6XjM/lUouUtwmCEwHxuNKN1UeKpmjn0pNQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731701441; a=rsa-sha256; cv=none; b=IRFbFBKzI0r3wJ31ykTllI7mLu0DZTZ7Pom/JZlrm8Mp2L4V36rYIM7Xmg4OKL71qcyEum CWC5md00x5wtpWnyFYtOJVrVFogpWakYZiYBUpZIfehlEOm8Yv6oxdzikJKCHy23FMtNxu 1lvo/jSCFC+3bwGPCyyK+MI2lEUuTIiIXll6zZH4U/gKRjvIZebWnw2IXv15l2e1PwuUd1 B64HpuA+zqem54gcIyZFP5WFJ4ZBibtTpuH0squjuYHN6nvzhTQ0cXI2t95Y+MWDx2cYXu 6rJA5npJ94r+yicFK0Jeh/EyhSjNnH5oB0eY+9bmwftqykO3TP5SzTTN8UzpjA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Xqp6n0TFpzcq8; Fri, 15 Nov 2024 20:10:41 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 4AFKAeJs069896; Fri, 15 Nov 2024 20:10:40 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 4AFKAetQ069893; Fri, 15 Nov 2024 20:10:40 GMT (envelope-from git) Date: Fri, 15 Nov 2024 20:10:40 GMT Message-Id: <202411152010.4AFKAetQ069893@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Ed Maste Subject: git: 48f3fcabea80 - main - mfc-candidates: Convert to Lua List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: emaste X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 48f3fcabea807b243aa73cacbd0cb1d32793b003 Auto-Submitted: auto-generated The branch main has been updated by emaste: URL: https://cgit.FreeBSD.org/src/commit/?id=48f3fcabea807b243aa73cacbd0cb1d32793b003 commit 48f3fcabea807b243aa73cacbd0cb1d32793b003 Author: Ed Maste AuthorDate: 2024-04-16 17:41:27 +0000 Commit: Ed Maste CommitDate: 2024-11-15 20:09:53 +0000 mfc-candidates: Convert to Lua d51c59002367 added a Lua script to process the lists of candidate and completed MFC commits to address sorting issues in the original shell implementation. Instead of having a mix of shell and Lua, just implement the entire tool in Lua. This is more maintainable and gives a reasonable improvement in performace. Reviewed by: imp Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D47416 --- tools/tools/git/candidatematch.lua | 74 ------------- tools/tools/git/mfc-candidates.lua | 218 +++++++++++++++++++++++++++++++++++++ tools/tools/git/mfc-candidates.sh | 138 +---------------------- 3 files changed, 220 insertions(+), 210 deletions(-) diff --git a/tools/tools/git/candidatematch.lua b/tools/tools/git/candidatematch.lua deleted file mode 100755 index 481c1f38fea1..000000000000 --- a/tools/tools/git/candidatematch.lua +++ /dev/null @@ -1,74 +0,0 @@ -#!/usr/libexec/flua - --- MFC candidate script utility - $0 from-file to-file --- --- from-file specifies hashes that exist only in the "MFC from" branch and --- to-file specifies the original hashes of commits already merged to the --- "MFC to" branch. - --- SPDX-License-Identifier: BSD-2-Clause --- Copyright 2024 The FreeBSD Foundation - --- Read a file and return its content as a table -local function read_file(filename) - local file = assert(io.open(filename, "r")) - local content = {} - for line in file:lines() do - table.insert(content, line) - end - file:close() - return content -end - --- Remove hashes from 'set1' list that are present in 'set2' list -local function set_difference(set1, set2) - local set2_values = {} - for _, value in ipairs(set2) do - set2_values[value] = true - end - - local result = {} - for _, value in ipairs(set1) do - if not set2_values[value] then - table.insert(result, value) - end - end - return result -end - --- Execute a command and print to stdout -local function exec_command(command) - local handle = io.popen(command) - local output = handle:read("a") - handle:close() - io.write(output) -end - --- Main function -local function main() - local from_file = arg[1] - local to_file = arg[2] - local exclude_file = arg[3] - - if not from_file or not to_file then - print("Usage: flua $0 from-file to-file") - return - end - - local from_hashes = read_file(from_file) - local to_hashes = read_file(to_file) - - local result_hashes = set_difference(from_hashes, to_hashes) - - if exclude_file then - exclude_hashes = read_file(exclude_file) - result_hashes = set_difference(result_hashes, exclude_hashes) - end - - -- Print the result - for _, hash in ipairs(result_hashes) do - exec_command("git show --pretty='%h %s' --no-patch " .. hash) - end -end - -main() diff --git a/tools/tools/git/mfc-candidates.lua b/tools/tools/git/mfc-candidates.lua new file mode 100755 index 000000000000..a8ac89bde327 --- /dev/null +++ b/tools/tools/git/mfc-candidates.lua @@ -0,0 +1,218 @@ +#!/usr/libexec/flua + +-- SPDX-License-Identifier: BSD-2-Clause +-- Copyright 2024 The FreeBSD Foundation + +-- MFC candidate search utility. Identify hashes that exist only in the +-- "MFC from" branch and do not have a corresponding "cherry picked from" +-- commit in the "MFC to" branch. + +-- Execute a command and return its output. A final newline is stripped, +-- similar to sh. +local function exec_command(command) + local handle = assert(io.popen(command)) + local output = handle:read("a") + handle:close() + if output:sub(-1) == "\n" then + return output:sub(1, -2) + end + return output +end + +-- Return a table of cherry-pick (MFC) candidates. +local function read_from(from_branch, to_branch, author, dirspec) + local command = "git rev-list --first-parent --reverse " + command = command .. to_branch .. ".." .. from_branch + if #author > 0 then + command = command .. " --committer \\<" .. author .. "@" + end + if dirspec then + command = command .. " " .. dirspec + end + if verbose > 1 then + print("Obtaining MFC-from commits using command:") + print(command) + end + local handle = assert(io.popen(command)) + local content = {} + for line in handle:lines() do + table.insert(content, line) + end + handle:close() + return content +end + +-- Return a table of original hashes of changes that have already been +-- cherry-picked (MFC'd). +local function read_to(from_branch, to_branch, dirspec) + local command = "git log " .. from_branch .. ".." .. to_branch + command = command .. " --grep 'cherry picked from'" + if dirspec then + command = command .. " " .. dirspec + end + if verbose > 1 then + print("Obtaining MFC-to commits using command:") + print(command) + end + local handle = assert(io.popen(command)) + local content = {} + for line in handle:lines() do + local hash = line:match("%(cherry picked from commit ([0-9a-f]+)%)") + if hash then + table.insert(content, hash) + end + end + handle:close() + return content +end + +-- Read a commit exclude file and return its content as a table. Comments +-- starting with # and text after a hash is ignored. +local function read_exclude(filename) + local file = assert(io.open(filename, "r")) + local content = {} + for line in file:lines() do + local hash = line:match("^%x+") + if hash then + -- Hashes are 40 chars; if less, expand short hash. + if #hash < 40 then + hash = exec_command( + "git show --pretty=%H --no-patch " .. hash) + end + table.insert(content, hash) + end + end + file:close() + return content +end + +--- Remove hashes from 'set1' list that are present in 'set2' list +local function set_difference(set1, set2) + local set2_values = {} + for _, value in ipairs(set2) do + set2_values[value] = true + end + + local result = {} + for _, value in ipairs(set1) do + if not set2_values[value] then + table.insert(result, value) + end + end + return result +end + +-- Global state +verbose = 0 + +local function params(from_branch, to_branch, author) + print("from: " .. from_branch) + print("to: " .. to_branch) + if #author > 0 then + print("author/committer: " .. author) + else + print("author/committer: ") + end +end + +local function usage(from_branch, to_branch, author) + local script_name = arg[0]:match("([^/]+)$") + print(script_name .. " [-ah] [-f from_branch] [-t to_branch] [-u user] [-X exclude_file] [path ...]") + print() + params(from_branch, to_branch, author) +end + +-- Main function +local function main() + local from_branch = "freebsd/main" + local to_branch = "" + local author = os.getenv("USER") or "" + local dirspec = nil + + local url = exec_command("git remote get-url freebsd") + local freebsd_repo = string.match(url, "[^/]+$") + freebsd_repo = string.gsub(freebsd_repo, ".git$", "") + if freebsd_repo == "ports" or freebsd_repo == "freebsd-ports" then + local year = os.date("%Y") + local month = os.date("%m") + local qtr = math.ceil(month / 3) + to_branch = "freebsd/" .. year .. "Q" .. qtr + elseif freebsd_repo == "src" or freebsd_repo == "freebsd-src" then + to_branch = "freebsd/stable/14" + -- If pwd is a stable or release branch tree, default to it. + local cur_branch = exec_command("git symbolic-ref --short HEAD") + if string.match(cur_branch, "^stable/") then + to_branch = cur_branch + elseif string.match(cur_branch, "^releng/") then + to_branch = cur_branch + local major = string.match(cur_branch, "%d+") + from_branch = "freebsd/stable/" .. major + end + else + print("pwd is not under a ports or src repository.") + return + end + + local do_help = false + local exclude_file = nil + local i = 1 + while i <= #arg and arg[i] do + local opt = arg[i] + if opt == "-a" then + author = "" + elseif opt == "-f" then + from_branch = arg[i + 1] + i = i + 1 + elseif opt == "-h" then + do_help = true + i = i + 1 + elseif opt == "-t" then + to_branch = arg[i + 1] + i = i + 1 + elseif opt == "-u" then + author = arg[i + 1] + i = i + 1 + elseif opt == "-v" then + verbose = verbose + 1 + elseif opt == "-X" then + exclude_file = arg[i + 1] + print ("-X not working") + i = i + 1 + else + break + end + i = i + 1 + end + + if do_help then + usage(from_branch, to_branch, author) + return + end + + if arg[i] then + dirspec = arg[i] + --print("dirspec = " .. dirspec) + -- XXX handle multiple dirspecs? + end + + if verbose > 0 then + params(from_branch, to_branch, author) + end + + local from_hashes = read_from(from_branch, to_branch, author, dirspec) + local to_hashes = read_to(from_branch, to_branch, dirspec) + + local result_hashes = set_difference(from_hashes, to_hashes) + + if exclude_file then + exclude_hashes = read_exclude(exclude_file) + result_hashes = set_difference(result_hashes, exclude_hashes) + end + + -- Print the result + for _, hash in ipairs(result_hashes) do + print(exec_command("git show --pretty='%h %s' --no-patch " .. hash)) + end +end + +main() diff --git a/tools/tools/git/mfc-candidates.sh b/tools/tools/git/mfc-candidates.sh index 7af4d29df67e..53fad909c91f 100644 --- a/tools/tools/git/mfc-candidates.sh +++ b/tools/tools/git/mfc-candidates.sh @@ -29,139 +29,5 @@ # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. -from_branch=freebsd/main -author="${USER}" - -# Get the FreeBSD repository -repo=$(basename "$(git remote get-url freebsd 2>/dev/null)" .git 2>/dev/null) - -if [ "${repo}" = "ports" -o "${repo}" = "freebsd-ports" ]; then - year=$(date '+%Y') - month=$(date '+%m') - qtr=$(((month-1) / 3 + 1)) - to_branch="freebsd/${year}Q${qtr}" -elif [ "${repo}" = "src" -o "${repo}" = "freebsd-src" ]; then - to_branch=freebsd/stable/14 - # If pwd is a stable or release branch tree, default to it. - cur_branch=$(git symbolic-ref --short HEAD 2>/dev/null) - case $cur_branch in - stable/*) - to_branch=$cur_branch - ;; - releng/*) - to_branch=$cur_branch - major=${cur_branch#releng/} - major=${major%.*} - from_branch=freebsd/stable/$major - esac -else - echo "pwd is not under a ports or src repository." - exit 0 -fi - -params() -{ - echo "from: $from_branch" - echo "to: $to_branch" - if [ -n "$author" ]; then - echo "author/committer: $author" - else - echo "author/committer: " - fi -} - -usage() -{ - echo "usage: $(basename $0) [-ah] [-f from_branch] [-t to_branch] [-u user] [-X exclude_file] [path ...]" - echo - params - exit 0 -} - -while getopts "af:ht:u:vX:" opt; do - case $opt in - a) - # All authors/committers - author= - ;; - f) - from_branch=$OPTARG - ;; - h) - usage - ;; - t) - to_branch=$OPTARG - ;; - u) - author=$OPTARG - ;; - v) - verbose=1 - ;; - X) - if [ ! -r "$OPTARG" ]; then - echo "Exclude file $OPTARG not readable" >&2 - exit 1 - fi - exclude_file=$OPTARG - ;; - esac -done -shift $(($OPTIND - 1)) - -if [ $verbose ]; then - params - echo -fi - -authorarg= -if [ -n "$author" ]; then - # Match user ID in the email portion of author or committer - authorarg="--committer <${author}@" -fi - -# Commits in from_branch after branch point -commits_from() -{ - git rev-list --first-parent --reverse $authorarg $to_branch..$from_branch "$@" -} - -# "cherry picked from" hashes from commits in to_branch after branch point -commits_to() -{ - git log $from_branch..$to_branch --grep 'cherry picked from' "$@" |\ - sed -E -n 's/^[[:space:]]*\(cherry picked from commit ([0-9a-f]+)\)[[:space:]]*$/\1/p' -} - -# Turn a list of short hashes (and optional descriptions) into a list of full -# hashes. -canonicalize_hashes() -{ - while read hash rest; do - case "${hash}" in - "#"*) continue ;; - esac - if ! git show --pretty=%H --no-patch $hash; then - echo "error parsing hash list" >&2 - exit 1 - fi - done | sort -} - -workdir=$(mktemp -d /tmp/find-mfc.XXXXXXXXXX) -from_list=$workdir/commits-from -to_list=$workdir/commits-to - -if [ -n "$exclude_file" ]; then - exclude_list=$workdir/commits-exclude - canonicalize_hashes < $exclude_file > $exclude_list -fi - -commits_from "$@" > $from_list -commits_to "$@" > $to_list - -/usr/libexec/flua $(dirname $0)/candidatematch.lua \ - $from_list $to_list $exclude_list - -rm -rf "$workdir" +# Backwards compatibility wrapper +/usr/libexec/flua $(dirname $0)/mfc-candidates.lua "$@"