ports/128754: [port infrastructure] implement master sites randomization

[Eygene Ryabinkin]

>Number:         128754
>Category:       ports
>Synopsis:       [port infrastructure] implement master sites randomization
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-ports-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Nov 10 16:00:09 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator:     Eygene Ryabinkin
>Release:        FreeBSD 7.1-PRERELEASE i386
>Organization:
Code Labs
>Environment:

System: FreeBSD 7.1-PRERELEASE i386

>Description:

Today I was hit by the very bad connectivity with twaren.net and
nchc.dl.sourceforge.net (the first site in the SF mirrors list in the
FreeBSD ports .mk files) is hosted by Taiwan REN.  So, I decided to
implement simple randomization that will enable to evenly distribute
the downloads between SF mirrors.

I had made it only for the SF mirrors, but it can be done for the other
well-known download locations as well.

>How-To-Repeat:

Look at ${PORTSDIR}/Mk/bsd.sites.mk: it uses statical assignment to
the MASTER_SITE_SOURCEFORGE.

>Fix:

The following patch works for me.  It places simple script rnd.awk to
the ${PORTSDIR}/Mk -- I don't think that this is the most appropriate
place to put it, but I can't think of another place now -- the script is
too ports-specific to put it into, for example, /usr/bin/.

--- 0001-Implement-randomization-of-main-SourceForge-sites.patch begins here ---
>From d48aa3982a5ca6192fad561462c3ba9da2b38cfa Mon Sep 17 00:00:00 2001
From: Eygene Ryabinkin <rea-fbsd at codelabs.ru>
Date: Mon, 10 Nov 2008 18:38:01 +0300

The reasoning behind this is simple: today I had experienced very bad
connectivity with twaren.net and it is the first mirror in the list.
So, for automated upgrades of ports that are using SourceForge mirrors,
the download was very time-consuming.  Such round-robin randomization
of the download sites helps to improve the situation.

Signed-off-by: Eygene Ryabinkin <rea-fbsd at codelabs.ru>
---
 Mk/bsd.commands.mk |    1 +
 Mk/bsd.sites.mk    |    9 ++++++---
 Mk/rnd.awk         |   14 ++++++++++++++
 3 files changed, 21 insertions(+), 3 deletions(-)
 create mode 100644 Mk/rnd.awk

diff --git a/Mk/bsd.commands.mk b/Mk/bsd.commands.mk
index aa8bde8..0b2a58b 100644
--- a/Mk/bsd.commands.mk
+++ b/Mk/bsd.commands.mk
@@ -69,6 +69,7 @@ PASTE?=		/usr/bin/paste
 PAX?=		/bin/pax
 PRINTF?=	/usr/bin/printf
 PW?=		/usr/sbin/pw
+RANDLIST?=	${AWK} -f ${PORTSDIR}/Mk/rnd.awk
 REALPATH?=	/bin/realpath
 RM?=		/bin/rm
 RMDIR?=		/bin/rmdir
diff --git a/Mk/bsd.sites.mk b/Mk/bsd.sites.mk
index 41b3f93..bd9a2a6 100644
--- a/Mk/bsd.sites.mk
+++ b/Mk/bsd.sites.mk
@@ -1188,20 +1188,23 @@ MASTER_SITE_SAVANNAH+= \
 
 .if !defined(IGNORE_MASTER_SITE_SOURCEFORGE)
 .for mirror in nchc kent easynews ufpr umn mesh heanet
-MASTER_SITE_SOURCEFORGE+= \
+MASTER_SITES_SOURCEFORGE+= \
 	http://${mirror}.dl.sourceforge.net/sourceforge/%SUBDIR%/
 .endfor
+MASTER_SITE_SOURCEFORGE != ${RANDLIST} ${MASTER_SITES_SOURCEFORGE}
 .endif
 
 # official sf.net mirrors that don't mirror all projects, check
 # http://prdownloads.sourceforge.net/%SUBDIR%/
 .if !defined(IGNORE_MASTER_SITE_SOURCEFORGE_EXTENDED)
 .for mirror in easynews switch puzzle belnet osdn ovh keihanna
-MASTER_SITE_SOURCEFORGE_EXTENDED+= \
+MASTER_SITES_SOURCEFORGE_EXTENDED+= \
 	http://${mirror}.dl.sourceforge.net/sourceforge/%SUBDIR%/
 .endfor
-MASTER_SITE_SOURCEFORGE_EXTENDED+= \
+MASTER_SITES_SOURCEFORGE_EXTENDED+= \
 	${MASTER_SITE_SOURCEFORGE}
+MASTER_SITE_SOURCEFORGE_EXTENDED != \
+	${RANDLIST} ${MASTER_SITES_SOURCEFORGE_EXTENDED}
 .endif
 
 .if !defined(IGNORE_MASTER_SITE_SOURCEFORGE_JP)
diff --git a/Mk/rnd.awk b/Mk/rnd.awk
new file mode 100644
index 0000000..fbfcad7
--- /dev/null
+++ b/Mk/rnd.awk
@@ -0,0 +1,14 @@
+BEGIN {
+# Need to drop a couple of initial rand() values: they tend
+# to be around 0.8 - 0.9, so for fairly small array lenght
+# they will produce identical values at the beginning.
+	srand(); rand(); rand(); rand(); rand();
+	for (i = ARGC - 1; i > 0; i--) {
+		j = int(10*ARGC*rand()) % i + 1;
+		if (j == i) continue;
+		t = ARGV[i]; ARGV[i] = ARGV[j]; ARGV[j] = t;
+	}
+
+	for (i = 1; i < ARGC; i++)
+		print ARGV[i];
+}
-- 
1.6.0.3
--- 0001-Implement-randomization-of-main-SourceForge-sites.patch ends here ---
>Release-Note:
>Audit-Trail:
>Unformatted: