Support for RfC 6249 - aka Metalinks - in ports?

[Raphael Eiselstein]

Hi,

I wonder if there will be support for RfC 6249 / Metalink in the FreeBSD
Ports infrastructure or someone is working / thinking about
implementing this.

From http://wiki.freebsd.org/Metalink
-------------------------------------------------------------
Metalink, an Open Standard, makes downloading easier by bundling the
various ways (FTP/HTTP/P2P) to get files into one format for easier
downloads. Simpler. Faster. More Reliable. Better.

Metalink is a great concept which might be added to the ports
infrastructure. Metalink files can be preprocessed in order to retain
big mirror lists in bsd.sites.mk. Or an XML guru can convert
bsd.sites.mk to an XML version and include it into metalinks in some
way. Stub metalinks can be easily created for each port with a simple
script.

At this time only www/aria2 has some Metalink support. Aria2 has a few
dependencies, but it's completely unnecessary to cram it into the base
system. FETCH_DEPENDS would work very nice.

Metalink support can be implemented simultaneously with P2P support,
which is long overdue. Aria2 fully supports Metalink 3.0 and other
downloaders will soon follow.

Metalink (last edited 2008-06-17 21:37:49 by localhost)
-------------------------------------------------------------

My approach would be some kind of http proxy which knows based on a
table of URLs from the ports all the mirror information (if there are
mirror sites) for given files or "content".

We already have MD5 and SHA256 checksums and sizes in distinfo, so we 
might enhance the fetch routines not to request a specific URL and 
filename but for a given checksum and writing the downloaded content 
to a known filename.

Instead of requesting
ftp://ftp.freebsd.org/pub/FreeBSD/ports/distfiles/aria2-1.10.9.tar.bz2
(or mirror sites) the ports might fetch
http://my-proxy.example.com/sha256/51319be3006545c442fbbedb476fa7ba2fd562e0468d9c573e7f5261ea610af5

and my-proxy.example.com would resolve this to a list of mirror URLs
using an implementation of RfC 6249 and maybe could itself be an
"dynamic ftp mirror" through some local caches (using squid-proxy or 
similar caches). 

    +------------------+
    |  FreeBSD Ports   |      Our ports (already have distinfo)
    +----+-------+-----+
         | fetch |            using fetch(1)
         +-------+
             |
             v
 +--------------------------+
 | http://localhost:8080/   | requesting checksums
 | sha256/51319be300...0af5 | from a local RfC 6249
 +--------------------------+ proxy/cache which might
     |        |       |       serve this from local
     v        v       v       cache or just will request
  +------+ +-----+ +------+   the file from multiple
  |  A   | |  B  | |  C   |   sources which have a copy
  +------+ +-----+ +------+   (of a copy (of a copy))
      \       |       /       of the requested file
       \      |      /        originally obtained from
        \     |     /         the master site or another
         +---------+          "primary source".
         | primary |
         | source  |
         +---------+

All we need to change in ports is a little enhancement which will make
fetch (optionally) requesting for the sha256 checksums instead of 
given filenames, enabled in /etc/make.conf

So, basically this would be a endoscopic surgery (=minimal impact) to 
the freebsd ports supporting optional requesting checksums instead of 
static URLs from a list.

Requesting static files parallel from multiple proxy caches might be 
implemented by parallel requesting byte ranges from different sources 
using 64kByte..256kByte blocks for example.  

Beside this, this approach wouldn't be limited to distfiles but may 
work for every kind of static files (tarballs, packages, ... ) by any 
open source project which is distributed through ftp mirrors. 

And all you need for this is

* a filename                    (we already have)
* a sha256 checksum             (we have)
* a rfc6249 proxy server        (3rd party component)
* a traditional HTTP/1.0 or HTTP/1.1 compliant tool (like fetch(1))

Does anyone like this approach? Does anyone work on (knows about) a 
concept or is developing something implementing rfc6249?

Regards
Raphael Eiselstein
-- 
Raphael Eiselstein <rabe at uugrn.org>               http://rabe.uugrn.org/
xmpp:freibyter at gmx.de  | https://www.xing.com/profile/Raphael_Eiselstein   
GnuPG:                E7B2 1D66 3AF2 EDC7 9828  6D7A 9CDA 3E7B 10CA 9F2D
.........|.........|.........|.........|.........|.........|.........|..
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
Url : http://lists.freebsd.org/pipermail/freebsd-ports/attachments/20110930/6ae15f7a/attachment.pgp