package distribution crisis - CDN needed

[Eberhard Moenkeberg]

Hi,

On Tue, 8 Apr 2008, Robert Watson wrote:
> On Mon, 7 Apr 2008, Pav Lucistnik wrote:
 
> > Okay the situation recently was that the mirrors had no chance keeping up
> > with all the package sets I've been uploading to ftp-master.
> >
> > We clearly need to move beyond rsync/cvsup synced ftp mirrors. This does not
> > scale.
> >
> > I do propose a creation of a CDN (Content Delivery Network), having these
> > features:
> 
> The success of a CDN is premised on the working set of active files being
> [significantly] smaller than the set of available files.  While I don't doubt
> that only a few packages are of interest to most users, for a CDN to provide a
> real benefit it needs to be the case that no users require most files.  The
> reason being that, if you have to transfer all or even most files every time,
> a CDN may well be a significantly less efficient way to do it than rsync. Has
> a working set analysis for our main mirrors been done that demonstrates that
> the actual working set is sufficiently smaller than the total set that a CDN
> would be more, rather than less, efficient?

Maybe it is interesting for you to have a look at 

http://www.poeml.de/~poeml/talks/redirector/

The SUSE team has a redirection mechanism at work on single file basis, 
and they also have experienced that it would be "enough" for the users if 
there are lots of mirrors with the "most requested" files plus some 
mirrors with all files.

It is all GPL, so you (even you) should be able to use the software.


Viele Grüße
Eberhard Mönkeberg (emoenke at gwdg.de, em at kki.org)