From nobody Sat Feb 26 07:37:07 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id BD20919DB33D; Sat, 26 Feb 2022 07:37:07 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K5JPb4sgKz3lmP; Sat, 26 Feb 2022 07:37:07 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1645861027; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=wTcOcU+M0a2cMlS/eGas15P5nHg0nHgjy2PFIxXaJ1E=; b=dortfRwaWk6Yqc+orlmjuWRqreVTmQUszp2uGDWlzb2YrbcUIQttOCozT001AUViF6t6nS ZrwpC2WvhEDyyFzjjkuEOcKlmTPFFco3KclsIo0yss8CnXTNYPjsk9QDbWk4m8xneAQssH hNahtz4mIfFoGnUEzppdlM+cIZaYeaPYmuA1vt57/eXwGs2glWkyRp4HxktYg4QxIxENWX lSY7bNIDFq8lp5NOfFTzQoxixQohiS/KIMyZx818nkK7XxcDqM1s1e1g9LQw5nCVNfYzV+ i2/sz0q9y9N32VSJuXAhs46v1No34aC2PNGISpzVleMloxdcFkoIaD7K/8r4Ow== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 839B479D4; Sat, 26 Feb 2022 07:37:07 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 21Q7b77H067023; Sat, 26 Feb 2022 07:37:07 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 21Q7b7SU067022; Sat, 26 Feb 2022 07:37:07 GMT (envelope-from git) Date: Sat, 26 Feb 2022 07:37:07 GMT Message-Id: <202202260737.21Q7b7SU067022@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Eugene Grosbein Subject: git: 259bedb8f6dc - stable/12 - fetch(1): merge several improvements for handling of soft failures List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: eugen X-Git-Repository: src X-Git-Refname: refs/heads/stable/12 X-Git-Reftype: branch X-Git-Commit: 259bedb8f6dce40fcabf228d2019337f70a60867 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1645861027; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=wTcOcU+M0a2cMlS/eGas15P5nHg0nHgjy2PFIxXaJ1E=; b=gC2ijlC9bEW/SJqYy1TWhQwJU8Kq6vuVirGaQpeGYbhbHuDejKA+pn5j7QsT73Z2w83g4b HH3TUhgqsAIUJ+v1VmbxKpFixIrlk6wbJiMc81BjCbFai4e2+0h1om9ggd4960gGi7F+2V BVQDyQQhRE6UpjPeePzShUy1SUH7C4uP7xS+7ZB4QqtxRGDJd/qlvkOJagCUHT8uIliKru GoaVabmvg/Vd0I27DejgLhyexyxeggQq+BL49nQ2H5yvGvjaH48678JEkhVKvm/7jKP1c2 iwj8HXyslpFn7ptj1hUfv5n+KafhDlWxWZsGgtKVp18DDdT5280B72lse/IWNQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1645861027; a=rsa-sha256; cv=none; b=y3DLPdFmY4qMD6d931b9Q9klM3Sp2lVA5+nQt8Hb8gdLfXGvhjGxbU4A664deHVF3p55Dd tD1XSVAdw0MXbg/YPqN7UO3wp1hAH4A5L540C0DzE+hzCTbGqPi+ASDvrpqaEgKPWwXUEU hQhFu3j1URCJLG0ZtZwDhx7hKNjj+s2fhta3J9UeOxLX/puYKRxeYPF/ISjOCmTzT5X+XM XOCHYy8QVwE1q5kN+RV5qO+buncb3X1JDbH7Lt6+5vmmnzrwshpNjemyUSXx5g7EFtvfAx QSemhfnd9ubNhd7lqXvKRqMm9rjJFaG8yT2KxgEc1jIB1zLCCl33GGltiskSjw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/12 has been updated by eugen: URL: https://cgit.FreeBSD.org/src/commit/?id=259bedb8f6dce40fcabf228d2019337f70a60867 commit 259bedb8f6dce40fcabf228d2019337f70a60867 Author: Eugene Grosbein AuthorDate: 2022-01-24 04:03:42 +0000 Commit: Eugene Grosbein CommitDate: 2022-02-26 07:36:00 +0000 fetch(1): merge several improvements for handling of soft failures Let "fetch -a" resume truncated transfer automatically to perform another attempt if it obtained some new data in previous one making progress. Do not consider HTTP 5XX errors as soft failures. Also, authorization errors should not be considered as soft failures. (cherry picked from commit e3bad5f7aa86a0911cf8d28395e7a29395739985) (cherry picked from commit a4efbe0d6da28eea0de4d414af25e7853ab8adfa) (cherry picked from commit bf599c03f09dea0f7e188e002b42d782af6841c3) (cherry picked from commit 08a2504a207c9302939bc0d1173fe44875e2a2e4) (cherry picked from commit 85f15576b423b9ad5b8a9e6dab3e71558ebe4335) --- usr.bin/fetch/fetch.c | 45 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 34 insertions(+), 11 deletions(-) diff --git a/usr.bin/fetch/fetch.c b/usr.bin/fetch/fetch.c index 22f7a1d2950b..73ee96541859 100644 --- a/usr.bin/fetch/fetch.c +++ b/usr.bin/fetch/fetch.c @@ -425,7 +425,7 @@ query_auth(struct url *URL) * Fetch a file */ static int -fetch(char *URL, const char *path) +fetch(char *URL, const char *path, int *is_http) { struct url *url; struct url_stat us; @@ -433,11 +433,11 @@ fetch(char *URL, const char *path) struct xferstat xs; FILE *f, *of; size_t size, readcnt, wr; - off_t count; + off_t count, size_prev; char flags[8]; const char *slash; char *tmppath; - int r; + int r, tries; unsigned timeout; char *ptr; @@ -475,6 +475,9 @@ fetch(char *URL, const char *path) strcpy(url->scheme, SCHEME_HTTP); } + /* for both of http and https */ + *is_http = strncmp(url->scheme, "http", 4) == 0; + /* common flags */ switch (family) { case PF_INET: @@ -537,6 +540,9 @@ fetch(char *URL, const char *path) goto success; } + tries = 1; +again: + r = 0; /* * If the -r flag was specified, we have to compare the local * and remote files, so we should really do a fetchStat() @@ -553,7 +559,7 @@ fetch(char *URL, const char *path) sb.st_size = -1; if (!o_stdout) { r = stat(path, &sb); - if (r == 0 && r_flag && S_ISREG(sb.st_mode)) { + if (r == 0 && (r_flag || tries > 1) && S_ISREG(sb.st_mode)) { url->offset = sb.st_size; } else if (r == -1 || !S_ISREG(sb.st_mode)) { /* @@ -568,6 +574,7 @@ fetch(char *URL, const char *path) goto failure; } } + size_prev = sb.st_size; /* start the transfer */ if (timeout) @@ -629,7 +636,7 @@ fetch(char *URL, const char *path) of = stdout; } else if (r_flag && sb.st_size != -1) { /* resume mode, local file exists */ - if (!F_flag && us.mtime && sb.st_mtime != us.mtime) { + if (!F_flag && us.mtime && sb.st_mtime != us.mtime && tries == 1) { /* no match! have to refetch */ fclose(f); /* if precious, warn the user and give up */ @@ -717,6 +724,8 @@ fetch(char *URL, const char *path) slash = path; else ++slash; + if(tmppath != NULL) + free(tmppath); asprintf(&tmppath, "%.*s.fetch.XXXXXX.%s", (int)(slash - path), path, slash); if (tmppath != NULL) { @@ -829,6 +838,13 @@ fetch(char *URL, const char *path) if (us.size != -1 && count < us.size) { warnx("%s appears to be truncated: %jd/%jd bytes", path, (intmax_t)count, (intmax_t)us.size); + if(!o_stdout && a_flag && count > size_prev) { + fclose(f); + if (w_secs) + sleep(w_secs); + tries++; + goto again; + } goto failure_keep; } @@ -898,7 +914,7 @@ main(int argc, char *argv[]) struct sigaction sa; const char *p, *s; char *end, *q; - int c, e, r; + int c, e, is_http, r; while ((c = getopt_long(argc, argv, @@ -1163,16 +1179,16 @@ main(int argc, char *argv[]) if (o_flag) { if (o_stdout) { - e = fetch(*argv, "-"); + e = fetch(*argv, "-", &is_http); } else if (o_directory) { asprintf(&q, "%s/%s", o_filename, p); - e = fetch(*argv, q); + e = fetch(*argv, q, &is_http); free(q); } else { - e = fetch(*argv, o_filename); + e = fetch(*argv, o_filename, &is_http); } } else { - e = fetch(*argv, p); + e = fetch(*argv, p, &is_http); } if (sigint) @@ -1184,11 +1200,18 @@ main(int argc, char *argv[]) if (e) { r = 1; if ((fetchLastErrCode + && fetchLastErrCode != FETCH_AUTH && fetchLastErrCode != FETCH_UNAVAIL && fetchLastErrCode != FETCH_MOVED && fetchLastErrCode != FETCH_URL && fetchLastErrCode != FETCH_RESOLV - && fetchLastErrCode != FETCH_UNKNOWN)) { + && fetchLastErrCode != FETCH_UNKNOWN + && (!is_http || ( + fetchLastErrCode != FETCH_PROTO + && fetchLastErrCode != FETCH_SERVER + && fetchLastErrCode != FETCH_TEMP + && fetchLastErrCode != FETCH_TIMEOUT + )))) { if (w_secs && v_level) fprintf(stderr, "Waiting %ld seconds " "before retrying\n", w_secs);