Downloading DISTFILES from multiple locations in parallel (torrent or ...)?

[Scot Hetzel]

On Sat, Nov 22, 2014 at 4:03 PM, Mikhail T. <mi+thun at aldan.algebra.com> wrote:
> Has anybody given the idea any thought? Though in addition to the
> "upstream" MASTER_SITES, there is a (fast) collection of FreeBSD
> mirrors, the downloads currently aren't happening in parallel -- they
> are serial...
>
> One solution would be for the downloader (fetch(1) or whatever) to try
> to open multiple connections -- to several possible sites -- and
> download different ranges of the same tarball from different locations
> in parallel thus affording the total download rate of the actually
> fastest mirror (almost), rather than the one first on the list.
>
> The alternative would use one of the existing torrent-clients, but we'd
> need to think of how to publish the torrent-information with the ports.
> It would seem, this second plan can go further even if the first would
> require fewer  changes to the existing infrastructure.
>
> Any thoughts?
>
I found this article on using the ftp/axel port to perform multiple
downloads for a file:

http://scratching.psybermonkey.net/2009/09/freebsd-download-ports-simultaneously.html

There is only one problem, the ports infrastructure only specifies one
download location for each file to the FETCH_CMD, to make it useful.
You would want to specify multiple download locations.

According to the axel web site:

http://axel.alioth.debian.org/

it can use multiple mirrors to get the file.  Someone would have to
re-code the the make fetch target so that multiple sites are specified
when axel is used:

axel -n 4 -a ftp://ftp1.freebsd.org/path/to/file.tgz
ftp://ftp.mirror.org/path/to/file.tgz

The above would open 4 connections, (2 connections to each site).

-- 
DISCLAIMER:

No electrons were maimed while sending this message. Only slightly bruised.