fetch: Resume errors, help!

[David Naylor]

Hi,

I have been having a very annoying problem with fetch and the ports distfiles 
fetching.  Living in South Africa internet access is not what it could be, as 
such I have to use a proxy to access the internet.  On to the problem:

When fetch tries to resume from an ftp site that does not support resuming (or 
fetch lacks the ability in these specific cases, or the proxy + fetch 
faulted) it appends to the file but restarts the transfer from 0.  This 
results in a file that is too big but does contain a complete copy of the 
file.  A solution is simple:
es = Expected size of the file
as = Actual size of the file
f = File name

# dd if=f of=f~ bs=$((as - es)) skip=1
# mv f~ f

This is very inconvenient and a problem when trying to download files in 
batch.  One gets interrupted and stops everything else.  As such I looked 
into fetch.c and tried to fix the problem.  See below for my patch.  A 
problem is that how to handle the -R flag (no overwrite file flag).  I have 
three solutions:

1) Ignore flag and overwrite the file anyway
2) Do not overwrite and abort
3) Specify above behaviour with an additional flag

Given that the ports and having many mirrors it is possible to find a site 
that does support resuming, and thus will save in bandwidth (important for 
me).  However it is possible that there are no alternatives or all sites do 
not support resume.  

In the patch the commented out section is solution 2, the implemented version 
is solution 1.  

Any advice will be welcome, once a satisfactory solution has been achieved I 
will submit a PR with the patch?  

Thank you
David

P.S. I've been using FreeBSD for a year or two however I am very new at 
writing patches for it, this will be my first so all help and mentoring will 
be very welcome.  

Patch file:
--- usr.bin/fetch/fetch.c.orig	2007-06-07 09:17:18.000000000 +0200
+++ usr.bin/fetch/fetch.c	2007-06-07 09:18:39.000000000 +0200
@@ -510,8 +510,36 @@
 				    (intmax_t)sb.st_size, (intmax_t)us.size);
 				goto failure;
 			}
+//			/* check fetched offset is as expected */
+//			if (url->offset != sb.st_size) {
+//				if (R_flag) {
+//				/* unable to resume */
+//					warnx("%s: unexpected offset, "
+//					    "expected %jd bytes, got %jd "
+//					    "bytes", (intmax_t)sb.st_size, 
+//					    (intmax_t)url->offset);
+//					goto failure_keep;
+//				} else if (url->offset != 0)
+//				/* if not we need to restart from scratch */
+//					goto restart;
+//			}
+//			if ((of = fopen(path, "a")) == NULL) {
+//				warn("%s: fopen()", path);
+//				goto failure;
+//			}
+//			/* check feteched offset is either as expected or 0 */
+			if (url->offset != sb.st_size || url->offset != 0)
+				/* if not we need to restart from scratch */
+				goto restart;
+			/*
+			 * NOTE: R_flag is ignored and local file is over 
+			 *	 written *not* deleted as it is the only way
+			 *	 to resume transfer of this file from this 
+			 *	 URL.
+			 */
 			/* we got it, open local file */
-			if ((of = fopen(path, "a")) == NULL) {
+			if ((of = url->offset == 0 ? fopen(path, "w") : 
+				  fopen(path, "a")) == NULL) {
 				warn("%s: fopen()", path);
 				goto failure;
 			}
@@ -536,6 +564,7 @@
 			goto success;
 	}
 
+restart:
 	if (of == NULL) {
 		/*
 		 * We don't yet have an output file; either this is a
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20070607/4f4b2bf2/attachment.pgp