OT: wget bug
Joe R. Jah
jjah at cloud.ccsf.cc.ca.us
Sun Jul 19 06:11:57 UTC 2009
On Sat, 18 Jul 2009, Andrew Brampton wrote:
> Date: Sat, 18 Jul 2009 18:09:54 +0100
> From: Andrew Brampton <brampton+freebsd at gmail.com>
> To: Joe R. Jah <jjah at cloud.ccsf.cc.ca.us>
> Cc: freebsd-questions at freebsd.org
> Subject: Re: OT: wget bug
> 2009/7/18 Joe R. Jah <jjah at cloud.ccsf.cc.ca.us>:
> > Thank you Andrew. Yes the server is truly returning 401. I have already
> > reconfigured wget to download everything regardless of their timestamp,
> > but it's a waste of bandwidth, because most of the site is unchanged.
> > Do you know of any workaround in wget, or an alternative tool to ONLY
> > download newer files by http?
> There are two ways to check if the file has been changed. One, read
> the time the file was last changed, or two, read the file and compare
> it to a old copy. Wget was obviously trying to do option 1 but this is
> denied by the remote server. You most likely could get it to do option
> 2, however by doing so you are wasting bandwidth downloading unchanged
> files just to check if they had been changed.
> If you have control over the remote webserver, then the simplest way
> to solve this problem is to configure the webserver not to return 401
> when wget sends the If-Modified-Since HTTP header. A better solution,
> again assuming you have control of the remote server, is to use
> "rsync" as it is designed for this kind of task.
> If you don't have control over the remote server, then you are stuck
> with your current solution.
Thank you Andrew.
_/ _/_/_/ _/ ____________ __o
_/ _/ _/ _/ ______________ _-\<,_
_/ _/ _/_/_/ _/ _/ ......(_)/ (_)
_/_/ oe _/ _/. _/_/ ah jjah at cloud.ccsf.cc.ca.us
More information about the freebsd-questions