package distribution crisis - CDN needed

[Pav Lucistnik]

Okay the situation recently was that the mirrors had no chance keeping
up with all the package sets I've been uploading to ftp-master.

We clearly need to move beyond rsync/cvsup synced ftp mirrors. This does
not scale.

I do propose a creation of a CDN (Content Delivery Network), having
these features:

- no mirroring of a complete package set! (Also no directory listings.)
  When client requests the file, and the file is not in the local cache,
  the file is downloaded from the upstream server and while it's being
  obtained, it's already being sent to the client. This is basically
  squid.

- if the file is present in the local cache, it's returned from local
  cache.

- local cache is invalidated when a new package set is available on
  an upstream server. Invalidating mechanism:
  option a) cronjob that polls upstream server every 5 minutes for a
            file that gives current package set IDs (pull)
  option b) master server sends notification to all mirrors to
            invalidate a package set (push)
  optimization: when package set was invalidated, don't delete old
  files, instead on next hit, verify timestamp against upstream server

- atomic package set uploads to master from pointyhat (probably having
  two directories that are switched over on master)

- everything runs over http

- default source of files for "pkg_add -r" command

The goal is to refresh a package set on a daily basis.


I don't know if we can use some existing software for this (Squid?
Apache mod_proxy?) or if we will need to put something new together.
Ideas?


-- 
Pav Lucistnik <pav at oook.cz>
              <pav at FreeBSD.org>
East or west, ~ is best.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: Toto je =?UTF-8?Q?digit=C3=A1ln=C4=9B?=
	=?ISO-8859-1?Q?_podepsan=E1?= =?UTF-8?Q?_=C4=8D=C3=A1st?=
	=?ISO-8859-1?Q?_zpr=E1vy?=
Url : http://lists.freebsd.org/pipermail/freebsd-hubs/attachments/20080407/51413a1c/attachment.pgp