Why FreeBSD fetch does not download a file via a proxy for HTTPS URLS (the same works fine for HTTP urls)

[chandra reddy]

Hi RW,

Thanks alot for your reply.

Do you mean to say "curl" also not using  a CONNECT to tunnel through to the
actual server?

How can I achieve downloading files HTTPS over a proxy?

Thanks
 <%20http://permalink.gmane.org/gmane.os.freebsd.devel.hackers/42588>
-Chandra

> Hi All,
>
> I am working on a project where i need to download a file via a proxy
> server using HTTPS protocol. I found that fetch does not work/support
> HTTPS requests over a proxy.

I just checked and neither do wget nor curl.

> I could overcome the above problem if I do the following change.
>
>                    1375:
> 1.58
> <http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libfetch/http.c#rev1.58>
> des      1376:        if (purl) { 1.51
> <http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libfetch/http.c#rev1.51>
> des      1377:                URL = purl;
>

I don't think that would work, presumably it would just cause an
attempt at an ssl connection to the proxy, followed by a GET request
for an https URL. https through a proxy is supposed to use a CONNECT to
tunnel through to the actual server.



On Thu, Feb 24, 2011 at 12:49 PM, chandra reddy <creddym at gmail.com> wrote:

> Hi All,
>
> I am working on a project where i need to download a file via a proxy
> server using HTTPS protocol. I found that fetch does not work/support HTTPS
> requests over a proxy.
>
> My setup would be like this:
>
>
>
> Intranet
> Internet
> -----------------------------------------------------------------------
> |                      https or  http                              |
>          https
> | Client m/cs -----------------------------> Porxy Server
> -------------------------------> Destination Server (or Download server)
> |                                                                      |
> -----------------------------------------------------------------------
>
>
> I can use https or http  protocol between Client and Proxy but only HTTPS
> is used between proxy and Destination server(or Download server) .
>
> I tried to use "squid" proxy as my proxy server and tried to download a
> file from my download server to Client m/c using FreeBSD "fetch" command.
> It fails to download a file via proxy for HTTPS requests Please note that
> Proxy setup is 100% correct and a web server (Apache) running fine.
> [I have tested it using my Mozilla browser on my PC].
>
> I have done the following:
>
> 1. *Download a file using HTTPS over a proxy server*
>
> #env HTTP_PROXY=http://<proxy-server-ip>:3128/ /usr/sbin/fetch -v -o
> /tmp/download.out 'https://<destination-server-ip>/index.htm'
>
> looking up <destination-server-ip>
>
> connecting to<destination-server-ip>:443
>
> connection established
>
> fetch: https://<destination-server-ip>/index.htm Authentication error
> Even I have tried this also and found the same error.
>
> #env HTTP_PROXY=https://<proxy-server-ip>:3128/ /usr/sbin/fetch -v -o
> /tmp/download.out 'https://<destination-serve-ip>/index.htm'
>
>
> My question is why it is not connected via "Proxy sever". It tries to
> connect directly. I could see that if I use HTTP protocol then it connects
> via proxy.
> Please see the logs here.
>
> 2. *Download a file using HTTP over a proxy server*
>
> #env HTTP_PROXY=http://<proxy-server-ip>:3128/ /usr/sbin/fetch -v -o
> /tmp/download.out 'http://<destination-server-ip>/index.htm'
>
> looking up <proxy-server-ip>
>
> connecting to <proxy-server-ip>:3128
>
> connection established
>
> requesting http://destination-server-ip/index.htm
> Even I have tried this also and found that works fine.
>
> #env HTTP_PROXY=https://<proxy-server-ip>:3128/ /usr/sbin/fetch -v -o
> /tmp/download.out 'http://<destination-serve-ip>/index.htm'
>
> I have debugged "fetch" and found that the following check is stopping
> HTTPS requests over a proxy.
>
> *http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libfetch/http.c
>
>                                      .OR.
>
> http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libfetch/http.c?annotate=1.78.2.5.4.1
>
> *
>
>                    1375:
> 1.58 <http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libfetch/http.c#rev1.58>      des      1376:        if (purl && strcasecmp(URL->scheme, SCHEME_HTTPS) != 0) {
> 1.51 <http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libfetch/http.c#rev1.51>      des      1377:                URL = purl;
>
>
>
> I could overcome the above problem if I do the following change.
>
>                    1375:
> 1.58 <http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libfetch/http.c#rev1.58>      des      1376:        if (purl) {
> 1.51 <http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libfetch/http.c#rev1.51>      des      1377:                URL = purl;
>
>
> I want to know why HTTPS over proxy is not working with "libfetch". I want
> to make it work how can do it?
>
> Thanks
> -Chandra
>
>


-- 
Thanks,
cr();
--------------------------------------------------------------------------------------------------------------------------
"Remote debugging a buggy debugger with a cross buggy debugger is a funny
thing"
--------------------------------------------------------------------------------------------------------------------------