package distribution crisis - CDN needed
pav at FreeBSD.org
Mon Apr 7 22:18:02 UTC 2008
Okay the situation recently was that the mirrors had no chance keeping
up with all the package sets I've been uploading to ftp-master.
We clearly need to move beyond rsync/cvsup synced ftp mirrors. This does
I do propose a creation of a CDN (Content Delivery Network), having
- no mirroring of a complete package set! (Also no directory listings.)
When client requests the file, and the file is not in the local cache,
the file is downloaded from the upstream server and while it's being
obtained, it's already being sent to the client. This is basically
- if the file is present in the local cache, it's returned from local
- local cache is invalidated when a new package set is available on
an upstream server. Invalidating mechanism:
option a) cronjob that polls upstream server every 5 minutes for a
file that gives current package set IDs (pull)
option b) master server sends notification to all mirrors to
invalidate a package set (push)
optimization: when package set was invalidated, don't delete old
files, instead on next hit, verify timestamp against upstream server
- atomic package set uploads to master from pointyhat (probably having
two directories that are switched over on master)
- everything runs over http
- default source of files for "pkg_add -r" command
The goal is to refresh a package set on a daily basis.
I don't know if we can use some existing software for this (Squid?
Apache mod_proxy?) or if we will need to put something new together.
Pav Lucistnik <pav at oook.cz>
<pav at FreeBSD.org>
East or west, ~ is best.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 195 bytes
Desc: Toto je =?UTF-8?Q?digit=C3=A1ln=C4=9B?=
Url : http://lists.freebsd.org/pipermail/freebsd-hubs/attachments/20080407/51413a1c/attachment.pgp
More information about the freebsd-hubs