/pub/FreeBSD or /FreeBSD

[Dominic Fandrey]

On 09/11/2010 13:48, Oliver Fromme wrote:
> 
> Dominic Fandrey wrote:
>  > I think I might have made a mistake here. I checked again and my
>  > estimate for half the servers was a) grossly exaggerated and b)
>  > the issue is more present over http.
>  > 
>  > Those don't work:
>  > http://ftp.de.freebsd.org/pub/FreeBSD/
>  > http://ftp3.de.freebsd.org/pub/FreeBSD/
>  > http://ftp4.de.freebsd.org/pub/FreeBSD/
>  > http://ftp6.de.freebsd.org/pub/FreeBSD/ (no HTTP at all)
>  > 
>  > These work:
>  > ftp://ftp.de.freebsd.org/pub/FreeBSD/
>  > ftp://ftp3.de.freebsd.org/pub/FreeBSD/
>  > ftp://ftp6.de.freebsd.org/pub/FreeBSD/
>  > 
>  > So, I have three distinct problems with half the German mirrors,
>  > two offer /pub/FreeBSD only through FTP, but not HTTP.
> 
> It is desirable that all FTP mirrors also offer the data
> via HTTP (and possibly also rsync or other protocols),
> but it is not a "must".  So, it is not surprising that
> some mirrors do not support HTTP.

3 of these 4 however have HTTP support. Shouldn't they follow the
expected pattern if they support HTTP? Probably their HTTP server
just doesn't follow Symlinks.

>  > One (ftp4.de) does not offer /pub/FreeBSD at all.
> 
> I consider that a bug.

Thank you.
 
> By the way, there's a mailing list specifically for the
> German mirrors:  de-bsd-hubs at de.freebsd.org  (posting in
> either German or English is ok).

I also encountered this with other mirrors, I just didn't bother
to test them all. Maybe I'll do a small probing script and provide
a complete list.
 
>  > And one does not offer HTTP access, so if I switched back to FTP,
>  > 3 quarters of my problem would disappear and I'd have 8 working
>  > mirrors instead of 5 out of 9.
>  > 
>  > So I should probably justify my use of HTTP:
>  > - Less latency (important for small downloads)
>  >   - 8-stable/All/automounter-1.4.3.tbz with the same connection:
>  >     HTTP: 0.20s, 0.20s, 0.20s
>  >     FTP: 0.39s, 0.62s, 0.39s, 0.39s (I'd throw the 0.62s away as a glitch)
>  >   - It needs 77 packages to install firefox, 70 of these are below 1.5m,
>  >     which is a small file by broadband standards, i.e. for 90% of
>  >     the packages latency is an important parameter
> 
> It depends very much on how the protocols are implemented
> by server and client.  If you use HTTP and *both* server
> and client support keep-alive, all files can be downloaded
> with a single connection.  Otherwise you need n connections
> (e.g. 77 connections for 77 files).  Some clients also open
> several connections in parallel.
> 
> With FTP, you have to use one control connection, and one
> data connection per file, so you have n+1 connections total
> (e.g. 78 connections for 77 files).  So it's not that much
> different from the non-keep-alive case.  I think FreeBSD's
> fetch(3) does not support keep-alive.

The client is fetch, my statements were based on that.

My application tries to always have one running download per
mirror (if there are enough files to be downloaded, at least).

I'd have to give up the single queue approach if I wanted to
use a keep-alive capability, if fetch supported it. And I
think single queue is the best approach to make the best out
of my internet connection.

> ...
> 
> Since FreeBSD's fetch(3) (used by sysinstall, pkg_add and
> the ports framework) does neither support keep-alive nor
> parallel downloads, the difference between HTTP and FTP is
> very small in practice.

I measured that it doesn't matter at all for downloads like
openoffice, this supports your assessment.


-- 
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?