/pub/FreeBSD or /FreeBSD

[Jason Harris]

On Tue, Nov 09, 2010 at 02:33:37PM +0100, Dominic Fandrey wrote:
> On 09/11/2010 13:48, Oliver Fromme wrote:

> > It depends very much on how the protocols are implemented
> > by server and client.  If you use HTTP and *both* server
> > and client support keep-alive, all files can be downloaded
> > with a single connection.  Otherwise you need n connections
> > (e.g. 77 connections for 77 files).  Some clients also open
> > several connections in parallel.

I have been testing aria2c (ports/www/aria2) for parallel-fetching
(1 connection per site, up to 32 sites) FreeBSD ports distfiles with
the following ~/.aria2/aria2.conf:

server-stat-if=server-stats
server-stat-of=server-stats
min-split-size=1M
allow-overwrite=true
uri-selector=adaptive
max-concurrent-downloads=32
timeout=15
connect-timeout=15
lowest-speed-limit=40K
remote-time=true
server-stat-timeout=86313600
split=32
enable-http-pipelining=true
use-head=true
event-poll=kqueue
file-allocation=none
log-level=notice

(NB:  This creates ./server-stats, usually in /usr/ports/distfiles,
with a few changes to /etc/make.conf and a patched ports/Mk/bsd.port.mk.)
It works quite well, often saturating my 4Mbit (down) link.

> My application tries to always have one running download per
> mirror (if there are enough files to be downloaded, at least).
> 
> I'd have to give up the single queue approach if I wanted to
> use a keep-alive capability, if fetch supported it. And I
> think single queue is the best approach to make the best out
> of my internet connection.

You can control instances of aria2c with XML-RPC (samples included
in the man page).  That would keep connections open and actively
downloading for as long as possible.  If you use one instance per
mirror, use a different server-stats and you can check each mirror's
overall performance when each aria2c finally exit()s.

> I measured that it doesn't matter at all for downloads like
> openoffice, this supports your assessment.

OpenOffice uses MirrorBrain ( http://www.mirrorbrain.org/users/ ),
which generates Metalink and BitTorrent files for built-in parallel
downloading.  Also, try "make fetch-url-list | xargs aria2c" in
ports/www/firefox to see what is possible with a small patch to
FreeBSD's existing ports infrastructure.

-- 
Jason Harris           |  PGP:  This _is_ PGP-signed, isn't it?
jharris at widomaker.com _|_ Got photons? (TM), (C) 2004
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 314 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-hubs/attachments/20101109/dceb791c/attachment.pgp