Why FreeBSD fetch does not download a file via a proxy for HTTPS URLS (the same works fine for HTTP urls)

[chandra reddy]

Hi All,

I am working on a project where i need to download a file via a proxy server
using HTTPS protocol. I found that fetch does not work/support HTTPS
requests over a proxy.

My setup would be like this:


    Intranet
Internet
-----------------------------------------------------------------------
|                      https or  http                              |
         https
| Client m/cs -----------------------------> Porxy Server
-------------------------------> Destination Server (or Download server)
|                                                                      |
-----------------------------------------------------------------------


I can use https or http  protocol between Client and Proxy but only HTTPS is
used between proxy and Destination server(or Download server) .

I tried to use "squid" proxy as my proxy server and tried to download a file
from my download server to Client m/c using FreeBSD "fetch" command.
It fails to download a file via proxy for HTTPS requests Please note that
Proxy setup is 100% correct and a web server (Apache) running fine.
[I have tested it using my Mozilla browser on my PC].

I have done the following:

1. *Download a file using HTTPS over a proxy server*

#env HTTP_PROXY=http://<proxy-server-ip>:3128/ /usr/sbin/fetch -v -o
/tmp/download.out 'https://<destination-server-ip>/index.htm'

looking up <destination-server-ip>

connecting to<destination-server-ip>:443

connection established

fetch: https://<destination-server-ip>/index.htm Authentication error
Even I have tried this also and found the same error.

#env HTTP_PROXY=https://<proxy-server-ip>:3128/ /usr/sbin/fetch -v -o
/tmp/download.out 'https://<destination-serve-ip>/index.htm'


My question is why it is not connected via "Proxy sever". It tries to
connect directly. I could see that if I use HTTP protocol then it connects
via proxy.
Please see the logs here.

2. *Download a file using HTTP over a proxy server*

#env HTTP_PROXY=http://<proxy-server-ip>:3128/ /usr/sbin/fetch -v -o
/tmp/download.out 'http://<destination-server-ip>/index.htm'

looking up <proxy-server-ip>

connecting to <proxy-server-ip>:3128

connection established

requesting http://destination-server-ip/index.htm
Even I have tried this also and found that works fine.

#env HTTP_PROXY=https://<proxy-server-ip>:3128/ /usr/sbin/fetch -v -o
/tmp/download.out 'http://<destination-serve-ip>/index.htm'

I have debugged "fetch" and found that the following check is stopping HTTPS
requests over a proxy.

*http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libfetch/http.c

                                     .OR.
http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libfetch/http.c?annotate=1.78.2.5.4.1

*

                   1375:
1.58 <http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libfetch/http.c#rev1.58>
     des      1376:        if (purl && strcasecmp(URL->scheme,
SCHEME_HTTPS) != 0) {
1.51 <http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libfetch/http.c#rev1.51>
     des      1377:                URL = purl;



I could overcome the above problem if I do the following change.

                   1375:
1.58 <http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libfetch/http.c#rev1.58>
     des      1376:        if (purl) {
1.51 <http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libfetch/http.c#rev1.51>
     des      1377:                URL = purl;


I want to know why HTTPS over proxy is not working with "libfetch". I want
to make it work how can do it?

Thanks
-Chandra