public_distfiles mirroring delay

[Erwin Lansing]

On Tue, Oct 09, 2007 at 01:42:04PM -0700, Eric Anholt wrote:
> In porting lame but useful software, I often have to roll up my own
> distfile because upstream doesn't bother ever releasing their code.  I
> stick the distfiles in ~/public_distfiles so it gets the
> load-distribution love, but it seems that none of the mirrors updates
> from there very frequently -- I usually end up not getting the file to
> fetch until the next day.  If I commit my port update before it gets
> distributed, I usually get email complaints from users that day.
> 
> Would it be possible for a single mirror that is in MASTER_SITE_LOCAL to
> sync local-distfiles more frequently, so that porters can commit
> local-distfile ports quicker after rolling them?

Given the asynchronous nature of the mirror updates, this is not a
trivial issue to solve, and has been on my mind before.  There are no
less than three steps required before the files show up on an ftp mirror.
First, you place the file in public_distfiles on freefall.  From there,
it has to be replicated to ftp-master(|.us|.eu).  Finally, the mirrors
have to pick it up from there.  For ftp.freebsd.org, I can promise that
the last step is done within hours (unless a very large dataset was just
uploaded to ftp-master).  Unless these files are made available directly
from freefall, adding even more traffic load to it, I don't really see
that this delay can be brought down even more and will always be several
hours.  My conclusion so far as been that the best way is to put the
file in public_distfiles the day before the commit.

Best,
-erwin


-- 
Erwin Lansing                                     http://droso.org
Security is like an onion.          (o_ _o)
It's made up of several layers   \\\_\   /_///    erwin at FreeBSD.org
And it makes you cry.            <____) (____>    erwin at aauug.dk
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-hubs/attachments/20071009/9e9b4f9f/attachment.pgp