bin/172451: [fetch] Not properly following redirects;
causes unnecessary manual intervention with ports installs, etc
Garrett Cooper
yanegomi at gmail.com
Sun Oct 7 13:20:12 UTC 2012
>Number: 172451
>Category: bin
>Synopsis: [fetch] Not properly following redirects; causes unnecessary manual intervention with ports installs, etc
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Sun Oct 07 13:20:11 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator: Garrett Cooper
>Release: 9.1-STABLE
>Organization:
EMC Isilon
>Environment:
>Description:
I have a fair amount of issues with fetch/libfetch and HTTP redirects. Some servers do it properly, others don't. A busted example (one of samba.org's HTTP servers) is as follows:
$ fetch -vv -AFpr http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
scheme: [http]
user: []
password: []
host: [us1.samba.org]
port: [0]
document: [/samba/ftp/stable/samba-3.6.8.tar.gz]
---> us1.samba.org:80
looking up us1.samba.org
connecting to us1.samba.org:80
requesting http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
>How-To-Repeat:
fetch -AFpr http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
>Fix:
Patch attached with submission follows:
Index: lib/libfetch/http.c
===================================================================
--- lib/libfetch/http.c (revision 241309)
+++ lib/libfetch/http.c (working copy)
@@ -1524,8 +1526,7 @@
/* try the provided URL first */
url = URL;
- /* if the A flag is set, we only get one try */
- n = noredirect ? 1 : MAX_REDIRECT;
+ n = MAX_REDIRECT;
i = 0;
e = HTTP_PROTOCOL_ERROR;
@@ -1772,6 +1774,15 @@
case hdr_location:
if (!HTTP_REDIRECT(conn->err))
break;
+ /*
+ * Don't redirect due to bad server
+ * paranoia.
+ */
+ if (noredirect &&
+ conn->err == HTTP_MOVED_TEMP) {
+ n = 1;
+ break;
+ }
if (new)
free(new);
if (verbose)
>Release-Note:
>Audit-Trail:
>Unformatted:
>>> GET /samba/ftp/stable/samba-3.6.8.tar.gz HTTP/1.1
>>> Host: us1.samba.org
>>> User-Agent: fetch libfetch/2.0
>>> Connection: close
>>>
<<< HTTP/1.1 301 Moved Permanently
<<< Date: Sun, 07 Oct 2012 11:14:28 GMT
<<< Server: Apache
<<< Location: http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
<<< Vary: Accept-Encoding
301 redirect to http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
scheme: [http]
user: []
password: []
host: [www.samba.org]
port: [0]
document: [/samba/ftp/stable/samba-3.6.8.tar.gz]
<<< Content-Length: 331
<<< Connection: close
content length: [331]
<<< Content-Type: text/html; charset=iso-8859-1
<<<
fetch: http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz: Moved Permanently
A working example with samba.org (without the -A flag):
$ fetch -vv -Fpr http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
scheme: [http]
user: []
password: []
host: [us1.samba.org]
port: [0]
document: [/samba/ftp/stable/samba-3.6.8.tar.gz]
---> us1.samba.org:80
looking up us1.samba.org
connecting to us1.samba.org:80
requesting http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
>>> GET /samba/ftp/stable/samba-3.6.8.tar.gz HTTP/1.1
>>> Host: us1.samba.org
>>> User-Agent: fetch libfetch/2.0
>>> Connection: close
>>>
<<< HTTP/1.1 301 Moved Permanently
<<< Date: Sun, 07 Oct 2012 11:25:32 GMT
<<< Server: Apache
<<< Location: http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
<<< Vary: Accept-Encoding
301 redirect to http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
scheme: [http]
user: []
password: []
host: [www.samba.org]
port: [0]
document: [/samba/ftp/stable/samba-3.6.8.tar.gz]
<<< Content-Length: 331
<<< Connection: close
content length: [331]
<<< Content-Type: text/html; charset=iso-8859-1
<<<
---> www.samba.org:80
looking up www.samba.org
connecting to www.samba.org:80
requesting http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
>>> GET /samba/ftp/stable/samba-3.6.8.tar.gz HTTP/1.1
>>> Host: www.samba.org
>>> User-Agent: fetch libfetch/2.0
>>> Connection: close
>>>
<<< HTTP/1.1 301 Moved Permanently
<<< Date: Sun, 07 Oct 2012 11:25:32 GMT
<<< Server: Apache
<<< Location: http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz
<<< Vary: Accept-Encoding
301 redirect to http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz
scheme: [http]
user: []
password: []
host: [ftp.samba.org]
port: [0]
document: [/pub/samba/stable/samba-3.6.8.tar.gz]
<<< Content-Length: 327
<<< Connection: close
content length: [327]
<<< Content-Type: text/html; charset=iso-8859-1
<<<
---> ftp.samba.org:80
looking up ftp.samba.org
connecting to ftp.samba.org:80
requesting http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz
>>> GET /pub/samba/stable/samba-3.6.8.tar.gz HTTP/1.1
>>> Host: ftp.samba.org
>>> User-Agent: fetch libfetch/2.0
>>> Connection: close
>>>
<<< HTTP/1.1 200 OK
<<< Date: Sun, 07 Oct 2012 11:25:32 GMT
<<< Server: Apache
<<< Last-Modified: Mon, 17 Sep 2012 07:13:13 GMT
<<< ETag: "125001d-207e10e-4c9e081b7b440"
last modified: [2012-09-17 07:13:13]
<<< Accept-Ranges: bytes
<<< Content-Length: 34070798
<<< Connection: close
content length: [34070798]
<<< Content-Type: application/x-gzip
<<< Content-Encoding: x-gzip
<<<
offset 0, length -1, size -1, clength 34070798
remote size / mtime: 34070798 / 1347865993
samba-3.6.8.tar.gz 2% of 32 MB 659 kBps^C
fetch: transfer interrupted
I think that the issue is that HTTP 3XX codes are blacklisted with the -A flag too liberally -- in particular 301, 302, and 307 should be followed per RFC-2616 [1]. The attached patch does that exactly (and also adds support for HTTP error code 305 -- us e proxy -- although scrubbing that before commit is fine if it should be in another bug).
FWIW that's what the documentation for fetch(1) suggests as well...
-A Do not automatically follow ``temporary'' (302) redirects.
Some broken Web sites will return a redirect instead of a
not-found error when the requested object does not exist.
fetch(3) doesn't have anything to say on the matter (and in fact doesn't document all of the supported flags)...
HTTP SCHEME
The fetchXGetHTTP(), fetchGetHTTP() and fetchPutHTTP() functions imple-
ment the HTTP/1.1 protocol. With a little luck, there is even a chance
that they comply with RFC2616 and RFC2617.
If the `d' (direct) flag is specified, fetchXGetHTTP(), fetchGetHTTP()
and fetchPutHTTP() will use a direct connection even if a proxy server is
defined.
If the `i' (if-modified-since) flag is specified, and the ims_time field
is set in struct url, then fetchXGetHTTP() and fetchGetHTTP() will send a
conditional If-Modified-Since HTTP header to only fetch the content if it
is newer than ims_time.
Since there seems to be no good way of implementing the HTTP PUT method
in a manner consistent with the rest of the fetch library, fetchPutHTTP()
is currently unimplemented.
After the patch is applied things function as I would expect:
$ fetch -vv -AFpr http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
scheme: [http]
user: []
password: []
host: [us1.samba.org]
port: [0]
document: [/samba/ftp/stable/samba-3.6.8.tar.gz]
---> us1.samba.org:80
looking up us1.samba.org
connecting to us1.samba.org:80
requesting http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
>>> GET /samba/ftp/stable/samba-3.6.8.tar.gz HTTP/1.1
>>> Host: us1.samba.org
>>> User-Agent: fetch libfetch/2.0
>>> Range: bytes=827392-
>>> Connection: close
>>>
<<< HTTP/1.1 301 Moved Permanently
<<< Date: Sun, 07 Oct 2012 12:36:27 GMT
<<< Server: Apache
<<< Location: http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
<<< Vary: Accept-Encoding
301 redirect to http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
scheme: [http]
user: []
password: []
host: [www.samba.org]
port: [0]
document: [/samba/ftp/stable/samba-3.6.8.tar.gz]
<<< Content-Length: 331
<<< Connection: close
content length: [331]
<<< Content-Type: text/html; charset=iso-8859-1
<<<
---> www.samba.org:80
looking up www.samba.org
connecting to www.samba.org:80
requesting http://www.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
>>> GET /samba/ftp/stable/samba-3.6.8.tar.gz HTTP/1.1
>>> Host: www.samba.org
>>> User-Agent: fetch libfetch/2.0
>>> Range: bytes=827392-
>>> Connection: close
>>>
<<< HTTP/1.1 301 Moved Permanently
<<< Date: Sun, 07 Oct 2012 12:36:28 GMT
<<< Server: Apache
<<< Location: http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz
<<< Vary: Accept-Encoding
301 redirect to http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz
scheme: [http]
user: []
password: []
host: [ftp.samba.org]
port: [0]
document: [/pub/samba/stable/samba-3.6.8.tar.gz]
<<< Content-Length: 327
<<< Connection: close
content length: [327]
<<< Content-Type: text/html; charset=iso-8859-1
<<<
---> ftp.samba.org:80
looking up ftp.samba.org
connecting to ftp.samba.org:80
requesting http://ftp.samba.org/pub/samba/stable/samba-3.6.8.tar.gz
>>> GET /pub/samba/stable/samba-3.6.8.tar.gz HTTP/1.1
>>> Host: ftp.samba.org
>>> User-Agent: fetch libfetch/2.0
>>> Range: bytes=827392-
>>> Connection: close
>>>
<<< HTTP/1.1 206 Partial Content
<<< Date: Sun, 07 Oct 2012 12:36:28 GMT
<<< Server: Apache
<<< Last-Modified: Mon, 17 Sep 2012 07:13:13 GMT
<<< ETag: "125001d-207e10e-4c9e081b7b440"
last modified: [2012-09-17 07:13:13]
<<< Accept-Ranges: bytes
<<< Content-Length: 33243406
<<< Content-Range: bytes 827392-34070797/34070798
content length: [33243406]
<<< Connection: close
content range: [827392-34070797/34070798]
<<< Content-Type: application/x-gzip
<<< Content-Encoding: x-gzip
<<<
offset 827392, length 33243406, size 34070798, clength 33243406
local size / mtime: 827392 / 1347865993
remote size / mtime: 34070798 / 1347865993
samba-3.6.8.tar.gz 100% of 32 MB 2301 kBps 00m00s
And removing the HTTP Error 302 clause I get behavior I would expect from fetch(1):
$ fetch -AFpr http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz
fetch: http://us1.samba.org/samba/ftp/stable/samba-3.6.8.tar.gz: Moved Permanently
References:
1. http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
More information about the freebsd-bugs
mailing list