[RFC] suport for fetching from local mirrors

[Ion-Mihai Tetcu]

On Wed, 10 Mar 2004 10:35:55 -0800
Brooks Davis <brooks at one-eyed-alien.net> wrote:

> On Wed, Mar 10, 2004 at 07:04:22PM +0200, Ion-Mihai Tetcu wrote:
> > Hi,
> > 
> > 
> > I wonder if there is any way to convince globally ``make fetch'' to try
> > use first other site that those in bsd.sites.mk (not with
> > MASTER_SITE_OVERRIDE, but overriding the master sites individually).
> > 
> > 
> > The reason for this (at least for .ro) is:
> > 
> > 1. the vast majority of the sites (apache, oo, *linux) have MAN mirrors
> > 2. many of them only partially mirror the master site
> > 3. or have a slightly different directory structure
> > 
> > This sites could be added to bsd.sites.mk, but:
> > 4. because of 2. and 3. above it would be a great idea
                                                  ^^^^^^^^ = wouldn't
> > 5 and the majority of mirrors admins wouldn't like abroad downloaders  as
> > the international bandwidth costs are very big . (10 to 50 compared to
> > us for example)
> > 
> > The benefits would be:
> > 6. unloading master sites and internet 
> > 7. faster download speed for users (and on large distfiles, like OO,
> > kde, etc. this would make a big difference esp. for "home" users with
> > low speed internet access).
> > 
> > This can be easily achieved by including a ``bsd.local_sites.mk'' in
> > bsd.ports.mk above line 2158 where bsd.sites.mk is included.
> > Something like:
> > 
> > # Local (MAN) master sites
> > .if exists(${PORTSDIR}/Mk/bsd.local_sites.mk)
> > .include "${PORTSDIR}/Mk/bsd.local_sites.mk"
> > .endif
> > 
> > The user will be responsible for creating and populating the file.
> 
> My first thought was that this was overkill, but upon further
> reflection, I like it.  

It took me about 3-4 hours to produce such a file last year. Of course
it wasn't very exhaustive, but it covered about 63% of the total size of
the distfiles for i386.

> I'm working toward doing most updates on systems
> at work via read-only nfs access to checked out copies of the ports
> tree, and this would let me eliminate about half of the /etc/make.conf
> configuration I use by having the ports collection contain the files
> that redirect to our local mirrors.

I have three cases:

1. Our local machines: the hole thing is very simple, as one of them is
cvsup-ing and fetching each 4th hour, and it's used as
MASTER_SITE_OVERRIDE for the others (being in a LAN this setup is just
easy and fast), on which the normal make install clean sequence also has
distclean to save space;

2. Our customers: some use our server - of course, when more of them
fetch in the same time the throughput drops drastically and maintaining
access rules is becoming to much time consuming, not to add that
maintaining a few versions for the distfiles is consuming space that
could be otherwise better use.

3. Our friends: being home users with lousy and / or expensive
international connections - they suffer the most.

> > Eventually an option for make.conf could be added, like:
> > LOCAL_FETCH_SITES= cc
> > where cc would be the country code. 
> > 
> > # Local (MAN) master sites
> > .if defined(LOCAL_FETCH_SITES)
> > .if exists(${PORTSDIR}/Mk/bsd.sites_${LOCAL_FETCH_SITES}.mk)
> > .include "${PORTSDIR}/Mk/bsd.sites_${LOCAL_FETCH_SITES}.mk"
> > .endif
> > .endif
> > 
> > The _cc files could be maintained by local user groups or something (I
> > would volunteer for _ro).
> 
> This seems like a reasonable idea to me.

Then perhaps _cc file plus local_sites, entries from local_sites being
on top (as they would be defined by the user - this would cover in your
case and some of my second case).

-- 
IOnut
Unregistered ;) FreeBSD user