Problem with pkg behind a chunking proxy

[Baptiste Daroussin]

On Tue, Mar 31, 2015 at 11:48:41PM +0200, sg-ball at laposte.net wrote:
> Some times ago I tried to install a FreeBSD 10.1 Release at work, where internal network is isolated from internet through a corporate proxy. And I could not make pkg work correctly.
> 
> I finally confirmed the problem was caused by the proxy using Transfer-Encoding: chunked instead of giving in first place a ContentLength header by using a python script acting as a proxy and forcing either chunked or not chunked response, and filed a bug report : https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198772 (got no followup till now ...)
> 
> I could dive in pkg 1.4.12 source code (thanks to the port) and found that the real cause was that in libpkg/fetch.c function pkg_fetch_file_to_fd used
> remote = fetchXGet(u, &st, sbuf_data(fetchOpts));
> to get remote file meta data in st (notably the file size) but did not test that st.size was not -1 (what it is when using Transfer-Encoding: chunked), so the following loop getting data up to st.size simply did not run.
> 
> So I could make some minimal changes to this file (fetch.c) to accept st.size == -1. Currently it simply writes "Fetching file: 0%" until the file is fully downloaded and then write full line with "100% size speed time". Of course it runs unchanged if it gets the file size through a ContentLength header.
> 
> It would certainly be better to trace the size all along the download, but it would require changes to libpkg/event.c as well, and I prefered to limit the changes for now. For anyone interested, I join the patch to this mail.
> 
> My questions now are :
> - is this the correct way to propose a patch ?
> - would it be better to propose the patch for pkg-devel ?
> - or should I use directly github a submit a pull request ?
> - or the feature must first be discussed here ?
> - or ...
> 
> I think this feature could be interesting since Transfer-Encoding: chunked is valid as HTTP 1.1 but I'm not used to submitting patches even if I've been using FreeBSD since release 3.x

First thank you for the patch! fixing this is on my TODO like forever.

Second sorry to not have followed up on the bug tracker, I am very busy in lot
of areas.

So usually the feature should be discussed in pkg at FreeBSD.org or in #pkgng
(freenode) or on github issue, I have to confess that I do track more often
github issue tracker than freebsd's bugzilla for pkg related issues. (which is
bad I should more track both).

Concerning your feature it is very welcome :) usually we do like pull request on
github. Note that you are just in time for pkg 1.5 which will be released soon :)

The preferred way is usually pull request, but I'm flexible if the submitter is
anti github then I can just grab patches.

That part of the code hasn't been modified since pkg 1.4 so the patch should
apply just fine.

Note that the bootstrap /usr/sbin/pkg in base suffer the same problems so that
patch will also apply there (maybe needed to be tuned a bit)

So as a conclusion first make a pull request on github, then once it is well
tested enough you can provide a patch for base (or I'll do) if so open a ticket
in the bug tracker for base and do not hesitate to harass me :)

Thanks again,
Bapt

> --- old/fetch.c	2015-02-13 20:34:33.000000000 +0100
> +++ new/fetch.c	2015-03-31 19:41:25.000000000 +0200
> @@ -618,30 +618,50 @@
>  
>  	pkg_emit_fetch_begin(url);
>  	pkg_emit_progress_start(NULL);
> -	while (done < sz) {
> -		int to_read = MIN(sizeof(buf), sz - done);
> +        if (sz > 0) {
> +		while (done < sz) {
> +			int to_read = MIN(sizeof(buf), sz - done);
> +
> +			pkg_debug(1, "Reading status: want read %d over %d, %d already done",
> +				to_read, sz, done);
> +			if ((r = fread(buf, 1, to_read, remote)) < 1)
> +				break;
> +
> +			if (write(dest, buf, r) != r) {
> +				pkg_emit_errno("write", "");
> +				retcode = EPKG_FATAL;
> +				goto cleanup;
> +			}
>  
> -		pkg_debug(1, "Reading status: want read %d over %d, %d already done",
> -			to_read, sz, done);
> -		if ((r = fread(buf, 1, to_read, remote)) < 1)
> -			break;
> +			done += r;
> +			pkg_debug(1, "Read status: %d over %d", done, sz);
>  
> -		if (write(dest, buf, r) != r) {
> -			pkg_emit_errno("write", "");
> +			pkg_emit_progress_tick(done, sz);
> +		}
> +
> +		if (done < sz) {
> +			pkg_emit_error("An error occurred while fetching package");
>  			retcode = EPKG_FATAL;
>  			goto cleanup;
>  		}
> -
> -		done += r;
> -		pkg_debug(1, "Read status: %d over %d", done, sz);
> -
> -		pkg_emit_progress_tick(done, sz);
> -	}
> -
> -	if (done < sz) {
> -		pkg_emit_error("An error occurred while fetching package");
> -		retcode = EPKG_FATAL;
> -		goto cleanup;
> +        }
> +	else {
> +		while ((r = fread(buf, 1, sizeof(buf), remote)) > 0) {
> +			if (write(dest, buf, r) != r) {
> +				pkg_emit_errno("write", "");
> +				retcode = EPKG_FATAL;
> +				goto cleanup;
> +			}
> +			done += r;
> +		}
> +		if (r != 0) {
> +			pkg_emit_error("An error occurred while fetching package");
> +			retcode = EPKG_FATAL;
> +			goto cleanup;
> +		}
> +		else {
> +			pkg_emit_progress_tick(done, done);
> +		}
>  	}
>  	pkg_emit_fetch_finished(url);
>  

> _______________________________________________
> freebsd-pkg at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-pkg
> To unsubscribe, send any mail to "freebsd-pkg-unsubscribe at freebsd.org"

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-pkg/attachments/20150401/ca7a6359/attachment.sig>