OT: wget bug

Andrew Brampton brampton+freebsd at gmail.com
Sat Jul 18 17:09:55 UTC 2009

2009/7/18 Joe R. Jah <jjah at cloud.ccsf.cc.ca.us>:
> Thank you Andrew.  Yes the server is truly returning 401.  I have already
> reconfigured wget to download everything regardless of their timestamp,
> but it's a waste of bandwidth, because most of the site is unchanged.
> Do you know of any workaround in wget, or an alternative tool to ONLY
> download newer files by http?

There are two ways to check if the file has been changed. One, read
the time the file was last changed, or two, read the file and compare
it to a old copy. Wget was obviously trying to do option 1 but this is
denied by the remote server. You most likely could get it to do option
2, however by doing so you are wasting bandwidth downloading unchanged
files just to check if they had been changed.

If you have control over the remote webserver, then the simplest way
to solve this problem is to configure the webserver not to return 401
when wget sends the If-Modified-Since HTTP header. A better solution,
again assuming you have control of the remote server, is to use
"rsync" as it is designed for this kind of task.

If you don't have control over the remote server, then you are stuck
with your current solution.


More information about the freebsd-questions mailing list