package distribution crisis - CDN needed

[Oliver Fromme]

Pav Lucistnik wrote:
 > Okay the situation recently was that the mirrors had no chance keeping
 > up with all the package sets I've been uploading to ftp-master.
 > 
 > We clearly need to move beyond rsync/cvsup synced ftp mirrors. This does
 > not scale.
 > 
 > I do propose a creation of a CDN (Content Delivery Network), having
 > these features:

First of all I agree with Robert's concerns.  We need to
find out if such a CDN is actually suitable according to
the download patterns of users.  It could well be that
it makes matters worse.

And second:  If a new solution is developed, it needs to
be portable and easy to install and maintain.  Remember
that some FreeBSD mirrors are run by non-FreeBSD people
on non-FreeBSD platforms.  I don't think we want to lose
those.

I've also read about at least one case where server and
client parts of a mirror ran on different machines, i.e.
the releases and package sets are synced on one machine
(which doesn't accept incoming connections for security
reasons) and the files are served to clients from a
different machine located in the DMZ of the site.  The
latter possibly can only accept incoming connections,
but can't make outgoing connections (policy).

We need to continue to support such setups.

Also, multi-level topology should still work, e.g. like
the following (4 levels):
 - ftpX.YY.freebsd.org sync from ftp.YY.freebsd.org
 - ftp.YY.freebsd.org syncs from ftp-master.eu.freebsd.org
 - ftp-master.eu.freebsd.org syncs from ftp-master.freebsd.org

Furthermore, you should still be able to browse the
packages with a standard FTP client.  I know several
people who are used to login to the nearest mirror,
cd to the packages' "All" directory and do things
like "dir gcc*", "mget xorg*" and so on.

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Geschäftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün-
chen, HRB 125758,  Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"Clear perl code is better than unclear awk code; but NOTHING
comes close to unclear perl code"  (taken from comp.lang.awk FAQ)