OT: wget bug

[Joe R. Jah]

On Sat, 18 Jul 2009, Andrew Brampton wrote:

> Date: Sat, 18 Jul 2009 18:09:54 +0100
> From: Andrew Brampton <brampton+freebsd at gmail.com>
> To: Joe R. Jah <jjah at cloud.ccsf.cc.ca.us>
> Cc: freebsd-questions at freebsd.org
> Subject: Re: OT: wget bug
>
> 2009/7/18 Joe R. Jah <jjah at cloud.ccsf.cc.ca.us>:
> > Thank you Andrew.  Yes the server is truly returning 401.  I have already
> > reconfigured wget to download everything regardless of their timestamp,
> > but it's a waste of bandwidth, because most of the site is unchanged.
> >
> > Do you know of any workaround in wget, or an alternative tool to ONLY
> > download newer files by http?
> >
>
> Joe,
> There are two ways to check if the file has been changed. One, read
> the time the file was last changed, or two, read the file and compare
> it to a old copy. Wget was obviously trying to do option 1 but this is
> denied by the remote server. You most likely could get it to do option
> 2, however by doing so you are wasting bandwidth downloading unchanged
> files just to check if they had been changed.
>
> If you have control over the remote webserver, then the simplest way
> to solve this problem is to configure the webserver not to return 401
> when wget sends the If-Modified-Since HTTP header. A better solution,
> again assuming you have control of the remote server, is to use
> "rsync" as it is designed for this kind of task.
>
> If you don't have control over the remote server, then you are stuck
> with your current solution.
>
> Andrew

Thank you Andrew.

Regards,

Joe
-- 
     _/   _/_/_/       _/              ____________    __o
     _/   _/   _/      _/         ______________     _-\<,_
 _/  _/   _/_/_/   _/  _/                     ......(_)/ (_)
  _/_/ oe _/   _/.  _/_/ ah        jjah at cloud.ccsf.cc.ca.us