Downloading DISTFILES from multiple locations in parallel (torrent or ...)?
swhetzel at gmail.com
Sun Nov 23 04:48:25 UTC 2014
On Sat, Nov 22, 2014 at 4:03 PM, Mikhail T. <mi+thun at aldan.algebra.com> wrote:
> Has anybody given the idea any thought? Though in addition to the
> "upstream" MASTER_SITES, there is a (fast) collection of FreeBSD
> mirrors, the downloads currently aren't happening in parallel -- they
> are serial...
> One solution would be for the downloader (fetch(1) or whatever) to try
> to open multiple connections -- to several possible sites -- and
> download different ranges of the same tarball from different locations
> in parallel thus affording the total download rate of the actually
> fastest mirror (almost), rather than the one first on the list.
> The alternative would use one of the existing torrent-clients, but we'd
> need to think of how to publish the torrent-information with the ports.
> It would seem, this second plan can go further even if the first would
> require fewer changes to the existing infrastructure.
> Any thoughts?
I found this article on using the ftp/axel port to perform multiple
downloads for a file:
There is only one problem, the ports infrastructure only specifies one
download location for each file to the FETCH_CMD, to make it useful.
You would want to specify multiple download locations.
According to the axel web site:
it can use multiple mirrors to get the file. Someone would have to
re-code the the make fetch target so that multiple sites are specified
when axel is used:
axel -n 4 -a ftp://ftp1.freebsd.org/path/to/file.tgz
The above would open 4 connections, (2 connections to each site).
No electrons were maimed while sending this message. Only slightly bruised.
More information about the freebsd-ports