bin/117277: [PATCH] fetch's resume mode doesn't verify that it actually got partial content

[Fabian Keil]

>Number:         117277
>Category:       bin
>Synopsis:       [PATCH] fetch's resume mode doesn't verify that it actually got partial content
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Oct 17 17:50:01 UTC 2007
>Closed-Date:
>Last-Modified:
>Originator:     Fabian Keil
>Release:        RELENG_7
>Organization:
>Environment:
FreeBSD TP51.local 7.0-PRERELEASE FreeBSD 7.0-PRERELEASE #1: Mon Oct 15 18:31:16 CEST 2007     fk at TP51.local:/usr/obj/usr/src/sys/THINKPAD  i386
>Description:
When resuming a file over HTTP, fetch(1) treats complete retransmits
like partial content and appends the whole response to the
already existing part that was fetched earlier.

The problem shows if the server doesn't support Range requests
or if the proxy strips out the Range header to prevent range
mismatches (which could happen if the proxy modified the first
response):

fk at TP51 /tank/fetch-tests $fetch -r http://10.0.0.1/BrooksDavis.EuroBSDCon.2007.avi
BrooksDavis.EuroBSDCon.2007.avi                30% of   92 MB 8440 kBps^C
fetch: transfer interrupted

fk at TP51 /tank/fetch-tests $fetch -r http://10.0.0.1/BrooksDavis.EuroBSDCon.2007.avi
BrooksDavis.EuroBSDCon.2007.avi               100% of   92 MB   17 MBps
fk at TP51 /tank/fetch-tests $ls -lh
total 123649
-rw-r--r--  1 fk  wheel   121M Sep 25 21:01 BrooksDavis.EuroBSDCon.2007.avi


>How-To-Repeat:
Resume a file from a HTTP server that doesn't support
Range requests or with a proxy that removes the Range
header.
>Fix:
While fetch(3) doesn't pass the HTTP status code to fetch(1),
the problem can be solved by checking the content offset.

If the offset is zero, the response contains the whole file
and the already existing part of it has to be overwritten.

With the attached patch it works as expected:

fk at TP51 /tank/fetch-tests $rm BrooksDavis.EuroBSDCon.2007.avi 
fk at TP51 /tank/fetch-tests $fetch -r http://10.0.0.1/BrooksDavis.EuroBSDCon.2007.avi
BrooksDavis.EuroBSDCon.2007.avi                58% of   92 MB   15 MBps^C
fetch: transfer interrupted

fk at TP51 /tank/fetch-tests $fetch -r http://10.0.0.1/BrooksDavis.EuroBSDCon.2007.avi
BrooksDavis.EuroBSDCon.2007.avi               100% of   92 MB   13 MBps
fk at TP51 /tank/fetch-tests $ls -lh
total 95081
-rw-r--r--  1 fk  wheel    93M Sep 25 21:01 BrooksDavis.EuroBSDCon.2007.avi


Patch attached with submission follows:

--- .zfs/snapshot/2007-10-15/usr.bin/fetch/fetch.c	2006-11-10 23:05:41.000000000 +0100
+++ usr.bin/fetch/fetch.c	2007-10-16 14:21:20.221581714 +0200
@@ -488,8 +488,11 @@
 	if (o_stdout) {
 		/* output to stdout */
 		of = stdout;
-	} else if (r_flag && sb.st_size != -1) {
-		/* resume mode, local file exists */
+	} else if (r_flag && sb.st_size != -1 && url->offset) {
+		/*
+		 * resume mode, local file exists and we
+		 * actually received partial content as requested
+		 */
 		if (!F_flag && us.mtime && sb.st_mtime != us.mtime) {
 			/* no match! have to refetch */
 			fclose(f);


>Release-Note:
>Audit-Trail:
>Unformatted: