From nobody Mon Jan 24 04:09:54 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 687F61970AAB; Mon, 24 Jan 2022 04:09:54 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JhxMk2V5hz4pfP; Mon, 24 Jan 2022 04:09:54 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642997394; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=hVODLWhe7Z+mTOjvc8Wg9QjRnLqVRTcvtvS9sj+hI5E=; b=dr3t7aZiRej5bnjgLdfn+ny0pSN/PrZ8T6GwxObOmfym2TEx88QEMCt+1duEZmNtWwRO2D I6WavHb31eiH4mRiE3R+IsqCRd09dfi5DCePCUaPXiQUty2iHthx52SMUum5Z160f4tNxn LGvy1akc5lgw3CFTM95AhX+js2VlAh6GVOytY4ElxYsJXxBszedxK4ka6zrK/iVx3/Ofxe RHhICtGeBVlmGut3/FJnhr8R+glfiwgHhnG9CUdidSURQx8LJAKJYFEacCWG4QnS3XbB9E BaitaGAvldnX5fujMbR+ebFpIivDL5z4ZwBSMOobHayaX7BDk7sGNl3bFviJQw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 358387606; Mon, 24 Jan 2022 04:09:54 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 20O49sjf019687; Mon, 24 Jan 2022 04:09:54 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 20O49swh019686; Mon, 24 Jan 2022 04:09:54 GMT (envelope-from git) Date: Mon, 24 Jan 2022 04:09:54 GMT Message-Id: <202201240409.20O49swh019686@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Eugene Grosbein Subject: git: e3bad5f7aa86 - main - fetch(1): process truncated transfer as soft failure List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: eugen X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: e3bad5f7aa86a0911cf8d28395e7a29395739985 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642997394; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=hVODLWhe7Z+mTOjvc8Wg9QjRnLqVRTcvtvS9sj+hI5E=; b=pW33BaZd2E0oYmxUu1jxvUef2L6xn8XMcmPUgMU75lzgIJ0p18+mxXBjosq278NjhojDd1 4N9jDXYJgDp4QpiNb1xRNGwwOOazWn44r9jpeuZNWxLjr1J8wulaMV78r/DzLFFDu9LiT0 uLmZA8YbP4ylQ9HGvhCViFnDzQWNUlzwUPsPeeBgOqz2jUUf970eO8eQesHxHnrFWfVfRY 3+FLbqYa+WLbDeC8NnfVShvXuRaLsjUAC3wDZCyB2AW0JR64cMLRFmi/cGgrWcD3qHqnSO G/NBPL4qLEGdq9pzBM2mXAyl77tjMZmTIHR6vlEq6jmdJ9gRjg+anNyPHAknYA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1642997394; a=rsa-sha256; cv=none; b=p0BriFKUPSs51tWcmftvUC0HwoRCaGAVa3x7HPkmYdqVcAxX6KtgrtjJ0Plr5Ipv+1uUOQ mpTW1zoEn3wb7oyfbNDvo31X5+U4IKVV8vXKpdmjDaAChXaHtvdHzFRqu8aaSS3emuN70l vPU/LgioKQcJ6dI0c/8n1W0L8kTcUYJ9+RTUxTdkJq56EHhAqM2EBdHSJXPM72pTY8Ga3Z y3JUPiHXl8hXEKxWOHRMJEYiUmIfMuCg9T0uC9eVIFnSiFuwqXjxK6C91n6N2L5o1waF2h RzYFb4HLAEy/O92YqbQWeSJQWEg9PEFQUXyAjePhJYD94PNyYhnXAov7fDCClw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by eugen: URL: https://cgit.FreeBSD.org/src/commit/?id=e3bad5f7aa86a0911cf8d28395e7a29395739985 commit e3bad5f7aa86a0911cf8d28395e7a29395739985 Author: Eugene Grosbein AuthorDate: 2022-01-24 04:03:42 +0000 Commit: Eugene Grosbein CommitDate: 2022-01-24 04:09:37 +0000 fetch(1): process truncated transfer as soft failure Let "fetch -a" resume truncated transfer automatically perform another attempt if it obtained some new data in previous one making progress. This makes it more robust against frequent but transient network failures. For example: => sqlite-src-3370200.zip doesn't seem to exist in /usr/ports/distfiles/. => Attempting to fetch https://www.sqlite.org/2022/sqlite-src-3370200.zip sqlite-src-3370200.zip 3% of 12 MB 45 kBps 04m24s fetch: sqlite-src-3370200.zip appears to be truncated: 524288/13145234 bytes sqlite-src-3370200.zip 10% of 12 MB 67 kBps 02m56s fetch: sqlite-src-3370200.zip appears to be truncated: 1327104/13145234 bytes sqlite-src-3370200.zip 28% of 12 MB 123 kBps 01m14s fetch: sqlite-src-3370200.zip appears to be truncated: 3735552/13145234 bytes sqlite-src-3370200.zip 54% of 12 MB 253 kBps 24s fetch: sqlite-src-3370200.zip appears to be truncated: 7176192/13145234 bytes sqlite-src-3370200.zip 62% of 12 MB 90 kBps 55s fetch: sqlite-src-3370200.zip appears to be truncated: 8241152/13145234 bytes sqlite-src-3370200.zip 82% of 12 MB 113 kBps 20s fetch: sqlite-src-3370200.zip appears to be truncated: 10862592/13145234 bytes sqlite-src-3370200.zip 12 MB 185 kBps 12s ===> Fetching all distfiles required by sqlite3-3.37.2,1 for building MFC after: 1 month --- usr.bin/fetch/fetch.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/usr.bin/fetch/fetch.c b/usr.bin/fetch/fetch.c index 22f7a1d2950b..3526e5d6c5c1 100644 --- a/usr.bin/fetch/fetch.c +++ b/usr.bin/fetch/fetch.c @@ -433,11 +433,11 @@ fetch(char *URL, const char *path) struct xferstat xs; FILE *f, *of; size_t size, readcnt, wr; - off_t count; + off_t count, size_prev; char flags[8]; const char *slash; char *tmppath; - int r; + int r, tries; unsigned timeout; char *ptr; @@ -537,6 +537,9 @@ fetch(char *URL, const char *path) goto success; } + tries = 1; +again: + r = 0; /* * If the -r flag was specified, we have to compare the local * and remote files, so we should really do a fetchStat() @@ -553,7 +556,7 @@ fetch(char *URL, const char *path) sb.st_size = -1; if (!o_stdout) { r = stat(path, &sb); - if (r == 0 && r_flag && S_ISREG(sb.st_mode)) { + if (r == 0 && (r_flag || tries > 1) && S_ISREG(sb.st_mode)) { url->offset = sb.st_size; } else if (r == -1 || !S_ISREG(sb.st_mode)) { /* @@ -568,6 +571,7 @@ fetch(char *URL, const char *path) goto failure; } } + size_prev = sb.st_size; /* start the transfer */ if (timeout) @@ -629,7 +633,7 @@ fetch(char *URL, const char *path) of = stdout; } else if (r_flag && sb.st_size != -1) { /* resume mode, local file exists */ - if (!F_flag && us.mtime && sb.st_mtime != us.mtime) { + if (!F_flag && us.mtime && sb.st_mtime != us.mtime && tries == 1) { /* no match! have to refetch */ fclose(f); /* if precious, warn the user and give up */ @@ -717,6 +721,8 @@ fetch(char *URL, const char *path) slash = path; else ++slash; + if(tmppath != NULL) + free(tmppath); asprintf(&tmppath, "%.*s.fetch.XXXXXX.%s", (int)(slash - path), path, slash); if (tmppath != NULL) { @@ -829,6 +835,13 @@ fetch(char *URL, const char *path) if (us.size != -1 && count < us.size) { warnx("%s appears to be truncated: %jd/%jd bytes", path, (intmax_t)count, (intmax_t)us.size); + if(!o_stdout && a_flag && us.size > size_prev) { + fclose(f); + if (w_secs) + sleep(w_secs); + tries++; + goto again; + } goto failure_keep; }