[Bug 219518] Fetch Command Times Out Using FTP Proxy

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219518

            Bug ID: 219518
           Summary: Fetch Command Times Out Using FTP Proxy
           Product: Base System
           Version: 11.0-STABLE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Many People
          Priority: ---
         Component: bin
          Assignee: freebsd-bugs at FreeBSD.org
          Reporter: johnllyon at gmail.com

Fetch (and I suspect libfetch) is unable to use an FTP proxy as specified by
the FTP_PROXY environment variable.  Attempting to download a file through an
FTP proxy (e.g. Squid) results in fetch timing out.  The error appears to be
that fetch is unable to interact with or properly negotiate with the proxy
server.  I suspect that this bug has not been noticed before because common
utilities that rely on fetch and proxies often use the HTTP protocol.

My setup:
    Client Machine -> Squid ver. 3.5.24 -> FTP Server

Following environment variables are set:

    setenv HTTP_PROXY http://192.168.1.1:3128
    setenv FTP_PROXY ftp://192.168.1.1:3128

Test Commands and Output:

Test Case 1 (to show Fetch Working with HTTP Proxy):

fetch -vvv http://www.cnn.com
scheme:   [http]
user:     []
password: []
host:     [www.cnn.com]
port:     [0]
document: [/]
scheme:   [http]
user:     []
password: []
host:     [192.168.1.1]
port:     [3128]
document: [/]
---> 192.168.1.1:3128
looking up 192.168.1.1
connecting to 192.168.1.1:3128
requesting http://www.cnn.com/
>>> GET http://www.cnn.com/ HTTP/1.1
>>> Host: www.cnn.com
>>> Accept: */*
>>> User-Agent: fetch libfetch/2.0
>>> Connection: close
>>>
<<< HTTP/1.1 200 OK
<<< access-control-allow-origin: *
<<< Cache-Control: max-age=60
<<< content-security-policy: default-src 'self' blob: https://*.cnn.com:*
http://*.cnn.com:* *.cnn.io:* *.cnn.net:* *.turner.com:* *.turner.io:*
*.ugdturner.com:* *.vgtf.net:*; script-src 'unsafe-eval' 'unsafe-inline' 'self'
*; style-src 'unsafe-inline' 'self' blob: *; child-src 'self' blob: *;
frame-src 'self' *; object-src 'self' *; img-src 'self' data: blob: *;
media-src 'self' blob: *; font-src 'self' data: *; connect-src 'self' *;
<<< Content-Type: text/html; charset=utf-8
<<< x-content-type-options: nosniff
<<< x-frame-options: SAMEORIGIN
<<< x-servedByHost: ::ffff:172.17.106.8
<<< x-xss-protection: 1; mode=block
<<< Fastly-Debug-Digest:
46be59e687681f2cbdc5286ab50024ed035dc360065b1aec7ce355bf418daeb9
<<< Content-Length: 139907
<<< Accept-Ranges: bytes
content length: [139907]
<<< Date: Thu, 25 May 2017 02:20:38 GMT
<<< Age: 21
<<< Set-Cookie: countryCode=US; Domain=.cnn.com; Path=/
<<< Set-Cookie: geoData=marietta|GA|30062|US|NA; Domain=.cnn.com; Path=/
<<< X-Served-By: cache-iad2146-IAD, cache-atl6230-ATL
<<< X-Cache: HIT, HIT
<<< X-Cache-Hits: 2, 4
<<< X-Timer: S1495678839.626481,VS0,VE0
<<< Vary: Accept-Encoding, Fastly-SSL, Fastly-SSL
<<< X-Cache: MISS from pfsense.johnllyon.com
<<< X-Cache-Lookup: HIT from pfsense.johnllyon.com:3128
<<< Via: 1.1 varnish, 1.1 varnish, ICAP/1.0 pfSense.johnllyon.com (C-ICAP/0.4.4
SquidClamav/Antivirus service ), 1.1 pfsense.johnllyon.com (squid/3.5.24)
<<< Connection: close
<<<
offset 0, length -1, size -1, clength 139907
local size / mtime: 139503 / 1495678456
remote size / mtime: 139907 / 0
www.cnn.com                                   100% of  136 kB   95 MBps 00m00s

Test Case 2 and 3 and Output (to show Fetch Timeout with Multiple FTP Servers
-- expected behavior is for requested text files to be displayed to stdout):

root at zeus# fetch -vvv ftp://ftp.netbsd.org/robots.txt
scheme:   [ftp]
user:     []
password: []
host:     [ftp.netbsd.org]
port:     [0]
document: [/robots.txt]
scheme:   [ftp]
user:     []
password: []
host:     [192.168.1.1]
port:     [3128]
document: [/]
---> 192.168.1.1:3128
looking up 192.168.1.1
connecting to 192.168.1.1:3128
fetch: transfer timed out


fetch -vvv
ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.0-RELEASE/MANIFEST
scheme:   [ftp]
user:     []
password: []
host:     [ftp.freebsd.org]
port:     [0]
document: [/pub/FreeBSD/releases/amd64/11.0-RELEASE/MANIFEST]
scheme:   [ftp]
user:     []
password: []
host:     [192.168.1.1]
port:     [3128]
document: [/]
---> 192.168.1.1:3128
looking up 192.168.1.1
connecting to 192.168.1.1:3128
fetch: transfer timed out

Test Case 4:

I setup my browser (Chrome) to use the Squid FTP proxy and navigated to the two
FTP servers used in Test Case 2 and Test Case 3.  In both cases, I could access
the FTP servers and files and I was presented with the Squid logo in browser to
notify me I was using the Squid proxy.  Hence the conclusion that the bug is in
fetch.

-- 
You are receiving this mail because:
You are the assignee for the bug.