From nobody Sat Feb 26 07:22:23 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 32A6C19D8C38; Sat, 26 Feb 2022 07:22:24 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K5J4c0v11z3klR; Sat, 26 Feb 2022 07:22:24 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1645860144; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=8jfhxrJjaUajigo8eXmIfu7jjCPXJITZXWk8GcuALzY=; b=nvyejOPU61w/OQyyw/LE3xHN5Lca3NFdx5XR4ood8aSsGi90J6s8PZqoqatyWLjt7WPE23 Ex7PQnGxfDXHsmwcrPj+9NAQP2LWukRi8vHphbQBjzJDbAvduql8h/uud9IddNYkIck2JN W/BWrzcoU9I3WxGAcZyOr19gmQoF/ulw187yxgk4e6c5yxWlLiGww6K8lGWlYx+crFzpOW e5nBzWF4gbH9LTU+078FC1IuTZfpsrulHvQpM4LLARBM5GccUYAK9yesXGdIENyf90WEkz K0aTqghoNMUdzr7ws6/ElCfpywYRFfiH7RiyZeu4l8qzzrwU39qZRanCZ0tkpA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id F29537648; Sat, 26 Feb 2022 07:22:23 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 21Q7MNV6052921; Sat, 26 Feb 2022 07:22:23 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 21Q7MNJ4052920; Sat, 26 Feb 2022 07:22:23 GMT (envelope-from git) Date: Sat, 26 Feb 2022 07:22:23 GMT Message-Id: <202202260722.21Q7MNJ4052920@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Eugene Grosbein Subject: git: 39d81d46aadf - stable/13 - fetch(1): merge several improvements for handling of soft failures List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: eugen X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 39d81d46aadf4d5ce1d702afe9e465808484ab18 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1645860144; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=8jfhxrJjaUajigo8eXmIfu7jjCPXJITZXWk8GcuALzY=; b=H26NyAkYw6Cv70azYATZNd5rrHGDiIrc3E4Ly+Sml2edZrSDPHNcsHzxe53YGLvle6URpC bIm7PeYrkEMxZPwya/GfhRwl9w3moVCnenBcLxV7cPAMaSGDxiX5OTzWJOK7wQUn8jceii ByKM1HEb4T8QcXvEmtv1dlfPB6smQI7ejzmECGnFyyk2V1jl/4KMRHf2bQvR2qMpnO50oG QnvQ911uXNWbp0VkPIQX4fMI8dcg0lqehpzoCEagQOZ8NKQ1OXQ4Eg70ATlge4EaQfRcBy GM1vRFYxnUTV6rPbkrcj8DRPbAsoWn5ImT9KpDyyIv5o28vXamUhELtNjCmC3A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1645860144; a=rsa-sha256; cv=none; b=OYPQHW4Ve2wdig1oZILElCXBsU82xejf+/Z/Bd5UCUsovfltGzGcYq/1yIlkFrjjsi0Z7L g+qQISml/pwmW4UCKQhW3wPawP0LfRrq/Z3cJ2Hr645pkKxwTSY91sazgPsL6eCeapDf74 fjCyFp2R9OI3wr9pJZisXf2OWwgOwK/ADcX6F4Wh+5o1euU4hifQXa8XmxOixGf7oWPHrE mQF1GzE9x2FE7pHFnvi+n5bUWN6gGIY/vtCRl1R6P7be6AA4x701lwL9v6Z84lEAvATYxC ls2QbyNZWYMCPW5eY6PUSs2IhnM06ueoXKiFX1kfdmGflGq7q5t4aAxz4NhtoQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by eugen: URL: https://cgit.FreeBSD.org/src/commit/?id=39d81d46aadf4d5ce1d702afe9e465808484ab18 commit 39d81d46aadf4d5ce1d702afe9e465808484ab18 Author: Eugene Grosbein AuthorDate: 2022-01-24 04:03:42 +0000 Commit: Eugene Grosbein CommitDate: 2022-02-26 07:19:19 +0000 fetch(1): merge several improvements for handling of soft failures Let "fetch -a" resume truncated transfer automatically to perform another attempt if it obtained some new data in previous one making progress. Do not consider HTTP 5XX errors as soft failures. Also, authorization errors should not be considered as soft failures. (cherry picked from commit e3bad5f7aa86a0911cf8d28395e7a29395739985) (cherry picked from commit a4efbe0d6da28eea0de4d414af25e7853ab8adfa) (cherry picked from commit bf599c03f09dea0f7e188e002b42d782af6841c3) (cherry picked from commit 08a2504a207c9302939bc0d1173fe44875e2a2e4) (cherry picked from commit 85f15576b423b9ad5b8a9e6dab3e71558ebe4335) --- usr.bin/fetch/fetch.c | 45 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 34 insertions(+), 11 deletions(-) diff --git a/usr.bin/fetch/fetch.c b/usr.bin/fetch/fetch.c index 22f7a1d2950b..73ee96541859 100644 --- a/usr.bin/fetch/fetch.c +++ b/usr.bin/fetch/fetch.c @@ -425,7 +425,7 @@ query_auth(struct url *URL) * Fetch a file */ static int -fetch(char *URL, const char *path) +fetch(char *URL, const char *path, int *is_http) { struct url *url; struct url_stat us; @@ -433,11 +433,11 @@ fetch(char *URL, const char *path) struct xferstat xs; FILE *f, *of; size_t size, readcnt, wr; - off_t count; + off_t count, size_prev; char flags[8]; const char *slash; char *tmppath; - int r; + int r, tries; unsigned timeout; char *ptr; @@ -475,6 +475,9 @@ fetch(char *URL, const char *path) strcpy(url->scheme, SCHEME_HTTP); } + /* for both of http and https */ + *is_http = strncmp(url->scheme, "http", 4) == 0; + /* common flags */ switch (family) { case PF_INET: @@ -537,6 +540,9 @@ fetch(char *URL, const char *path) goto success; } + tries = 1; +again: + r = 0; /* * If the -r flag was specified, we have to compare the local * and remote files, so we should really do a fetchStat() @@ -553,7 +559,7 @@ fetch(char *URL, const char *path) sb.st_size = -1; if (!o_stdout) { r = stat(path, &sb); - if (r == 0 && r_flag && S_ISREG(sb.st_mode)) { + if (r == 0 && (r_flag || tries > 1) && S_ISREG(sb.st_mode)) { url->offset = sb.st_size; } else if (r == -1 || !S_ISREG(sb.st_mode)) { /* @@ -568,6 +574,7 @@ fetch(char *URL, const char *path) goto failure; } } + size_prev = sb.st_size; /* start the transfer */ if (timeout) @@ -629,7 +636,7 @@ fetch(char *URL, const char *path) of = stdout; } else if (r_flag && sb.st_size != -1) { /* resume mode, local file exists */ - if (!F_flag && us.mtime && sb.st_mtime != us.mtime) { + if (!F_flag && us.mtime && sb.st_mtime != us.mtime && tries == 1) { /* no match! have to refetch */ fclose(f); /* if precious, warn the user and give up */ @@ -717,6 +724,8 @@ fetch(char *URL, const char *path) slash = path; else ++slash; + if(tmppath != NULL) + free(tmppath); asprintf(&tmppath, "%.*s.fetch.XXXXXX.%s", (int)(slash - path), path, slash); if (tmppath != NULL) { @@ -829,6 +838,13 @@ fetch(char *URL, const char *path) if (us.size != -1 && count < us.size) { warnx("%s appears to be truncated: %jd/%jd bytes", path, (intmax_t)count, (intmax_t)us.size); + if(!o_stdout && a_flag && count > size_prev) { + fclose(f); + if (w_secs) + sleep(w_secs); + tries++; + goto again; + } goto failure_keep; } @@ -898,7 +914,7 @@ main(int argc, char *argv[]) struct sigaction sa; const char *p, *s; char *end, *q; - int c, e, r; + int c, e, is_http, r; while ((c = getopt_long(argc, argv, @@ -1163,16 +1179,16 @@ main(int argc, char *argv[]) if (o_flag) { if (o_stdout) { - e = fetch(*argv, "-"); + e = fetch(*argv, "-", &is_http); } else if (o_directory) { asprintf(&q, "%s/%s", o_filename, p); - e = fetch(*argv, q); + e = fetch(*argv, q, &is_http); free(q); } else { - e = fetch(*argv, o_filename); + e = fetch(*argv, o_filename, &is_http); } } else { - e = fetch(*argv, p); + e = fetch(*argv, p, &is_http); } if (sigint) @@ -1184,11 +1200,18 @@ main(int argc, char *argv[]) if (e) { r = 1; if ((fetchLastErrCode + && fetchLastErrCode != FETCH_AUTH && fetchLastErrCode != FETCH_UNAVAIL && fetchLastErrCode != FETCH_MOVED && fetchLastErrCode != FETCH_URL && fetchLastErrCode != FETCH_RESOLV - && fetchLastErrCode != FETCH_UNKNOWN)) { + && fetchLastErrCode != FETCH_UNKNOWN + && (!is_http || ( + fetchLastErrCode != FETCH_PROTO + && fetchLastErrCode != FETCH_SERVER + && fetchLastErrCode != FETCH_TEMP + && fetchLastErrCode != FETCH_TIMEOUT + )))) { if (w_secs && v_level) fprintf(stderr, "Waiting %ld seconds " "before retrying\n", w_secs);