svn commit: r44849 - head/en_US.ISO8859-1/articles/hubs
Gavin Atkinson
gavin at FreeBSD.org
Sat May 17 02:50:17 UTC 2014
Author: gavin
Date: Sat May 17 02:50:17 2014
New Revision: 44849
URL: http://svnweb.freebsd.org/changeset/doc/44849
Log:
Start to bring this up-to-date:
- Drop all references to providing CVSup mirrors
- Drop all references to providing anonymous CVS mirrors
- Drop all references to mirroring using CVSup, instead use rsync
- Update the recommended arguments to rsync in the examples
- Updatre size estimates for the individual parts
- Change one reference to CVS to instead mention SVN
Note that we are still not accepting applications for new mirrors
of this style.
Modified:
head/en_US.ISO8859-1/articles/hubs/article.xml
Modified: head/en_US.ISO8859-1/articles/hubs/article.xml
==============================================================================
--- head/en_US.ISO8859-1/articles/hubs/article.xml Fri May 16 19:28:26 2014 (r44848)
+++ head/en_US.ISO8859-1/articles/hubs/article.xml Sat May 17 02:50:17 2014 (r44849)
@@ -57,7 +57,7 @@
and degree of completeness you want to mirror, a huge
amount of disk space may be consumed. Also keep in mind
that <emphasis>official</emphasis> mirrors are probably required to be
- complete. The CVS repository and the web pages should
+ complete. The web pages should
always be mirrored completely. Also note that the
numbers stated here are reflecting the current
state (at &rel2.current;-RELEASE/&rel.current;-RELEASE). Further development and
@@ -67,10 +67,9 @@
Here are some approximate figures:
</para>
<itemizedlist>
- <listitem><para>Full FTP Distribution: 1.1 TB</para></listitem>
- <listitem><para>CVS repository: 5.4 GB</para></listitem>
- <listitem><para>CTM deltas: 3.2 GB</para></listitem>
- <listitem><para>Web pages: 463 MB</para></listitem>
+ <listitem><para>Full FTP Distribution: 1.4 TB</para></listitem>
+ <listitem><para>CTM deltas: 10 GB</para></listitem>
+ <listitem><para>Web pages: 1GB</para></listitem>
</itemizedlist>
<para>
The current disk usage of FTP Distribution can be found at
@@ -113,7 +112,7 @@
</para>
<para>
For a moderately visited site that offers
- <application>Rsync</application>, you might
+ <application>rsync</application>, you might
consider a current CPU with around 800MHz - 1 GHz,
and at least 512MB RAM. This is probably the
minimum you want for an <emphasis>official</emphasis>
@@ -127,7 +126,7 @@
</para>
<para>
You also want to consider a fast disk subsystem.
- Operations on the CVS repository require a fast
+ Operations on the SVN repository require a fast
disk subsystem (RAID is highly advised). A SCSI
controller that has a cache of its own can also
speed up things since most of these services incur a
@@ -181,17 +180,9 @@
</listitem>
<listitem><para><package>ftp/twoftpd</package>: As above.</para></listitem>
<listitem><para><package>ftp/vsftpd</package>: The <quote>very secure</quote> ftpd.</para></listitem>
- <listitem>
- <para><package>ftp/wu-ftpd</package>: The ftpd from Washington
- University. It has become infamous, because of the huge
- amount of security issues that have been found in it.
- If you do choose to use this software be sure to
- keep it up to date.
- </para>
- </listitem>
</itemizedlist>
- <para>FreeBSD's <application>ftpd</application>, <application>proftpd</application>,
- <application>wu-ftpd</application> and maybe <application>ncftpd</application>
+ <para>FreeBSD's <application>ftpd</application>, <application>proftpd</application>
+ and maybe <application>ncftpd</application>
are among the most commonly used FTPds.
The others do not have a large userbase among mirror sites. One
thing to consider is that you may need flexibility in limiting
@@ -260,109 +251,6 @@
</listitem>
</itemizedlist>
</sect3>
- <sect3 xml:id="mirror-serv-cvsup">
- <title>CVSup (desired for CVS repository)</title>
- <para>
- <application>CVSup</application> is a very efficient way of distributing files.
- It works similar to <application>rsync</application>, but was specially designed for
- use with CVS repositories. If you want to offer the
- FreeBSD CVS repository, you really want to consider
- offering it via <application>CVSup</application>. It is possible to offer
- the CVS repository via <application>AnonCVS</application>, FTP,
- <application>rsync</application> or HTTP, but
- people would benefit much more from <application>CVSup</application> access.
- <application>CVSup</application> was developed by &a.jdp.email;.
- It is a bit tricky to install on non-FreeBSD platforms,
- since it is written in Modula-3 and therefore requires
- a Modula-3 environment. &a.jdp; has built a
- stripped down version of M3 that is sufficient to
- run <application>CVSup</application>, and can be installed much easier.
- See <link xlink:href="http://www.polstra.com/projects/freeware/ezm3/">Ezm3</link>
- for details. Related ports are:</para>
-
- <itemizedlist>
- <listitem>
- <para><package>net/cvsup</package>: The native CVSup port (client and server)
- which requires <package>lang/ezm3</package> now.</para>
- </listitem>
- <listitem>
- <para><package>net/cvsup-mirror</package>: The CVSup mirror kit, which requires
- <package>net/cvsup-without-gui</package>, and configures it mirror-ready. Some
- site administrators may want a different setup though.
- </para>
- </listitem>
- </itemizedlist>
-
- <para>There are a few more like
- <package>net/cvsup-without-gui</package> you might want to have
- a look at. If you prefer a static binary package, take a look
- <link xlink:href="http://people.FreeBSD.org/~jdp/s1g/">here</link>.
- This page still refers to the S1G bug that was present
- in <application>CVSup</application>. Maybe
- John will set up a generic download-site to get
- static binaries for various platforms.
- </para>
- <para>
- It is possible to use <application>CVSup</application> to offer
- any kind of fileset, not just CVS repositories,
- but configuration can be complex.
- <application>CVSup</application> is known to eat some CPU on both the server and the
- client, since it needs to compare lots of files.
- </para>
- </sect3>
- <sect3 xml:id="mirror-anoncvs">
- <title>AnonCVS (optional for CVS repository)</title>
- <para>
- If you have the CVS repository, you may want to offer
- anonymous CVS access. A short warning first:
- There is not much demand for it,
- it requires some experience, and you need to know
- what you are doing.
- </para>
- <para>
- Generally there are two ways
- to access a CVS repository remotely: via
- <emphasis>pserver</emphasis> or via <command>ssh</command>
- (we do not consider <command>rsh</command>).
- For anonymous access, <emphasis>pserver</emphasis> is
- very well suited, but some still offer <command>ssh</command>
- access as well. There is a custom crafted
- <link xlink:href="ftp://ftp.FreeBSD.org/pub/FreeBSD/development/FreeBSD-CVS/anoncvs.shar">wrapper</link>
- in the CVS repository, to be used as a login-shell for the
- anonymous ssh account. It does a chroot, and therefore
- requires the CVS repository to be available under the
- anonymous user's home-directory. This may not be possible
- for all sites. If you just offer <emphasis>pserver</emphasis>
- this restriction does not apply, but you may run with
- more security risks. You do not need to install any special
- software, since &man.cvs.1; comes with
- FreeBSD. You need to enable access via <command>inetd</command>,
- so add an entry into your <filename>/etc/inetd.conf</filename>
- like this:</para>
- <programlisting>
-cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --allow-root=/home/ncvs pserver
- </programlisting>
- <para>See the manpage for details of the options. Also see the CVS <emphasis>info</emphasis>
- page about additional ways to make sure access is read-only.
- It is advised that you create an unprivileged account,
- preferably called <systemitem class="username">anoncvs</systemitem>.
- Also you need to create a file <filename>passwd</filename>
- in your <filename>/home/ncvs/CVSROOT</filename> and assign a
- CVS password (empty or <literal>anoncvs</literal>) to that user.
- The directory <filename>/anoncvstmp</filename> is a special
- purpose memory based file system. It is not required but
- advised since &man.cvs.1; creates a shadow directory
- structure in your <filename>/tmp</filename> which is
- not used after the operation but slows things
- dramatically if real disk operations are required.
- Here is an excerpt from <filename>/etc/fstab</filename>,
- how to set up such a MFS:</para>
- <programlisting>
-/dev/da0s1b /anoncvstmp mfs rw,-s=786432,-b=4096,-f=512,-i=560,-c=3,-m=0,nosuid 0 0
- </programlisting>
- <para>This is (of course) tuned a lot, and was suggested by &a.jdp.email;.
- </para>
- </sect3>
</sect2>
</sect1>
<sect1 xml:id="mirror-howto">
@@ -374,8 +262,8 @@ cvspserver stream tcp nowait root /usr/b
the various parts of FreeBSD, what tools to use,
and where to mirror from.
</para>
- <sect2 xml:id="mirror-ftp">
- <title>FTP</title>
+ <sect2 xml:id="mirror-ftp-rsync">
+ <title>Mirroring the FTP site</title>
<para>
The FTP area is the largest amount of data that
needs to be mirrored. It includes the <emphasis>distribution
@@ -388,39 +276,8 @@ cvspserver stream tcp nowait root /usr/b
for various FreeBSD versions,
and various architectures.
</para>
- <sect3 xml:id="mirror-ftp-ftp">
- <title>With FTP mirror</title>
<para>
- You can use a <application>FTP mirror</application>
- program to get the files. Some of the most commonly used are:</para>
- <itemizedlist>
- <listitem><para><package>ftp/mirror</package></para></listitem>
- <listitem><para><package>ftp/ftpmirror</package></para></listitem>
- <listitem><para><package>ftp/emirror</package></para></listitem>
- <listitem><para><package>ftp/spegla</package></para></listitem>
- <listitem><para><package>ftp/omi</package></para></listitem>
- <listitem><para><package>ftp/wget</package></para></listitem>
- </itemizedlist>
-
- <para><package>ftp/mirror</package> was very popular, but seemed
- to have some drawbacks, as it is written in &man.perl.1;,
- and had real problems with mirroring large
- directories like a FreeBSD site. There are rumors that
- the current version has fixed this by allowing
- a different algorithm for comparing
- the directory structure to be specified.
- </para>
- <para>
- In general FTP is not really good for mirroring. It transfers
- the whole file if it has changed, and does
- not create a single data stream which would benefit from
- a large TCP congestion window.
- </para>
- </sect3>
- <sect3 xml:id="mirror-ftp-rsync">
- <title>With rsync</title>
- <para>
- A better way to mirror the FTP area is <application>rsync</application>.
+ The best way to mirror the FTP area is <application>rsync</application>.
You can install the port <package>net/rsync</package> and then use
rsync to sync with your upstream host.
<application>rsync</application> is already mentioned
@@ -440,7 +297,7 @@ cvspserver stream tcp nowait root /usr/b
</para>
</note>
<para>A command line to mirror FreeBSD might look like:</para>
- <screen>&prompt.user; <userinput>rsync -vaz --delete ftp4.de.FreeBSD.org::FreeBSD/ /pub/FreeBSD/</userinput>
+ <screen>&prompt.user; <userinput>rsync -vaHz --delete rsync://ftp4.de.FreeBSD.org/FreeBSD/ /pub/FreeBSD/</userinput>
</screen>
<para>Consult the documentation for <application>rsync</application>,
which is also available at
@@ -453,188 +310,15 @@ cvspserver stream tcp nowait root /usr/b
want to set up a script framework that calls such a command
via &man.cron.8;.
</para>
- </sect3>
- <sect3 xml:id="mirror-ftp-cvsup">
- <title>With CVSup</title>
- <para>
- A few sites, including the one-and-only <systemitem class="fqdomainname">ftp-master.FreeBSD.org</systemitem>
- even offer <application>CVSup</application> to mirror the contents of
- the FTP space. You need to install a <application>CVSup</application>
- client, preferably from the port <package>net/cvsup</package>.
- (Also reread <xref linkend="mirror-serv-cvsup"/>.)
- A sample <filename>supfile</filename> suitable for <systemitem class="fqdomainname">ftp-master.FreeBSD.org</systemitem>
- looks like this:</para>
- <programlisting>
- #
- # FreeBSD archive supfile from master server
- #
- *default host=ftp-master.FreeBSD.org
- *default base=/usr
- *default prefix=/pub
- #*default release=all
- *default delete use-rel-suffix
- *default umask=002
-
- # If your network link is a T1 or faster, comment out the following line.
- #*default compress
-
- FreeBSD-archive release=all preserve
- </programlisting>
-
- <para>It seems <application>CVSup</application> would be the best
- way to mirror the archive in terms of efficiency, but
- it is only available from few sites.</para>
- <note xml:id="mirror-cvsup-s-option">
- <para>
- Please have look at the <application>CVSup</application> documentation
- like &man.cvsup.1; and consider using the <option>-s</option>
- option. This reduces I/O operations by assuming the
- recorded information about each file is correct.</para>
- </note>
- </sect3>
- </sect2>
- <sect2 xml:id="mirror-cvs">
- <title>Mirroring the CVS repository</title>
- <para>There are various ways to mirror the CVS repository.
- <application>CVSup</application> is the most common method.</para>
-
- <sect3 xml:id="mirror-cvs-cvsup">
- <title>Using CVSup</title>
- <para>
- <application>CVSup</application> is described in some
- detail in <xref linkend="mirror-serv-cvsup"/> and <xref linkend="mirror-ftp-cvsup"/>.
- </para>
- <para>It is very easy to setup a
- <application>CVSup</application> mirror. Installing
- <package>net/cvsup-mirror</package> will
- make sure all of the needed programs are installed and then
- gather all the needed information to configure the mirror.</para>
- <note>
- <para>
- Please do not forget to consider the hint
- mentioned in <link linkend="mirror-cvsup-s-option">this note</link>
- above.
- </para>
- </note>
- </sect3>
- <sect3 xml:id="mirror-cvs-other">
- <title>Using other methods</title>
- <para>
- Using other methods than <application>CVSup</application> is
- generally not recommended. We describe them in short here
- anyway. Since most sites offer the CVS repository as
- part of the FTP fileset under the path
- <filename>/pub/FreeBSD/development/FreeBSD-CVS</filename>,
- the following methods could be used.</para>
- <itemizedlist>
- <listitem><para><application>FTP</application></para></listitem>
- <listitem><para><application>Rsync</application></para></listitem>
- <listitem><para><application>HTTP</application></para></listitem>
- </itemizedlist>
-
- <important>
- <para>AnonCVS cannot be used to mirror the CVS repository
- since CVS does not allow you to access the repository
- itself, only checked out versions of the modules.</para>
- </important>
- </sect3>
</sect2>
<sect2 xml:id="mirror-www">
<title>Mirroring the WWW pages</title>
<para>
- The best way is to check out the <emphasis>www</emphasis>
- distribution from CVS. If you have a local mirror of the
- CVS repository, it is as easy as:</para>
- <screen>&prompt.user; <userinput>cvs -d /home/ncvs co www</userinput></screen>
- <para>and a <emphasis>cronjob</emphasis>, that calls <command>cvs up -d -P</command>
- on a regular basis, maybe just after your repository was updated.
- Of course, the files need to remain in a directory available
- for public WWW access. The installation and configuration of a
- web server is not discussed here.
- </para>
-
- <para>
- If you do not have a local repository, you can use
- <application>CVSup</application> to maintain an <quote>up to date copy</quote>
- of the www pages. A sample supfile can be found in
- <filename>/usr/share/examples/cvsup/www-supfile</filename> and
- could look like this:</para>
- <programlisting>
- #
- # WWW module supfile for FreeBSD
- #
- *default host=cvsup3.de.FreeBSD.org
- *default base=/usr
- *default prefix=/usr/local
- *default release=cvs tag=.
- *default delete use-rel-suffix
-
- # If your network link is a T1 or faster, comment out the following line.
- *default compress
-
- # This collection retrieves the www/ tree of the FreeBSD repository
- www
- </programlisting>
- <para>
- Using <package>ftp/wget</package> or other web-mirror tools is
- not recommended.
- </para>
- <sect3 xml:id="mirror-www-doc">
- <title>Mirroring the FreeBSD documentation</title>
- <para>
- Since the documentation is referenced a lot from the
- web pages, it is recommended that you mirror the
- FreeBSD documentation as well. However, this is not
- as trivial as the www-pages alone.
- </para>
- <para>
- First of all, you should get the doc sources,
- again preferably via <application>CVSup</application>.
- Here is a corresponding sample supfile:</para>
- <programlisting>
- #
- # FreeBSD documentation supfile
- #
- *default host=cvsup3.de.FreeBSD.org
- *default base=/usr
- *default prefix=/usr/share
- *default release=cvs tag=.
- *default delete use-rel-suffix
-
- # If your network link is a T1 or faster, comment out the following line.
- #*default compress
-
- # This will retrieve the entire doc branch of the FreeBSD repository.
- # This includes the handbook, FAQ, and translations thereof.
- doc-all
- </programlisting>
- <para>
- Then you need to install a couple of ports.
- You are lucky, there is a meta-port:
- <package>textproc/docproj</package> to do the work
- for you. You need to set up some
- environment variables, like
- <literal>SGML_CATALOG_FILES</literal>.
- Also have a look at your <filename>/etc/make.conf</filename>
- (copy <filename>/usr/share/examples/etc/make.conf</filename> if
- you do not have one), and look at the
- <literal>DOC_LANG</literal> variable.
- Now you are probably ready to run <command>make</command>
- in your doc directory (<filename>/usr/share/doc</filename>
- by default) and build the documentation.
- Again you need to make it accessible for your web server
- and make sure the links point to the right location.</para>
- <important>
- <para>
- The building of the documentation, as well as lots
- of side issues, is documented itself in the
- <link xlink:href="&url.books.fdp-primer;">&os; Documentation
- Project Primer</link>.
- Please read this piece of documentation, especially if you
- have problems building the documentation.
- </para>
- </important>
- </sect3>
+ The FreeBSD website should only be mirrored via
+ <application>rsync</application>.</para>
+ <para>A command line to mirror the FreeBSD web site might look like:</para>
+ <screen>&prompt.user; <userinput>rsync -vaHz --delete rsync://bit0.us-west.freebsd.org/FreeBSD-www-data/ /usr/local/www/</userinput>
+ </screen>
</sect2>
<sect2 xml:id="mirror-how-often">
<title>How often should I mirror?</title>
@@ -683,7 +367,6 @@ cvspserver stream tcp nowait root /usr/b
Here are some recommended schedules:</para>
<itemizedlist>
<listitem><para>FTP fileset: daily</para></listitem>
- <listitem><para>CVS repository: hourly</para></listitem>
<listitem><para>WWW pages: daily</para></listitem>
</itemizedlist>
</sect2>
@@ -708,7 +391,7 @@ cvspserver stream tcp nowait root /usr/b
<emphasis>N</emphasis> is a number,
telling that the host would be the <emphasis>Nth</emphasis>
mirror in that country.
- (Same applies to <systemitem>cvsupN.CC.FreeBSD.org</systemitem>,
+ (Same applies to
<systemitem>wwwN.CC.FreeBSD.org</systemitem>, etc.)
There are mirrors with no <emphasis>CC</emphasis> part.
These are the mirror sites that are very well connected and
@@ -812,9 +495,7 @@ cvspserver stream tcp nowait root /usr/b
that.
</para>
<para>
- There are three master sites for the FTP fileset and
- one for the CVS repository (the web pages and docs are
- obtained from CVS, so there is no need for master).
+ There is one master site for the FTP fileset.
</para>
<sect4 xml:id="mirror-where-master-ftp">
<title>ftp-master.FreeBSD.org</title>
@@ -823,11 +504,9 @@ cvspserver stream tcp nowait root /usr/b
</para>
<para>
<systemitem>ftp-master.FreeBSD.org</systemitem> provides
- <application>rsync</application> and <application>CVSup</application>
+ <application>rsync</application>
access, in addition to FTP.
- Refer to <xref linkend="mirror-ftp-rsync"/> and
- <xref linkend="mirror-ftp-cvsup"/> how to access
- via these protocols.
+ Refer to <xref linkend="mirror-ftp-rsync"/>.
</para>
<para>
Mirrors are also encouraged to allow <application>rsync</application>
@@ -835,32 +514,6 @@ cvspserver stream tcp nowait root /usr/b
<emphasis>Tier-1</emphasis>-mirrors.
</para>
</sect4>
- <sect4 xml:id="mirror-where-master-cvsup">
- <title>cvsup-master.FreeBSD.org</title>
- <para>
- This is the master site for the CVS repository.
- </para>
- <para>
- <systemitem>cvsup-master.FreeBSD.org</systemitem> provides
- <application>CVSup</application> access only.
- See <xref linkend="mirror-cvs-cvsup"/> for details.
- </para>
- <para>
- To get access, you need to contact the &a.cvsup-master;.
- Make sure you read the
- <link xlink:href="http://people.FreeBSD.org/~jdp/cvsup-access/">FreeBSD CVSup Access Policy</link>
- first!
- </para>
- <para>
- Set up the required authentication by following
- <link xlink:href="http://people.FreeBSD.org/~jdp/cvpasswd/">these
- instructions</link>. Make sure you specify the server as
- <systemitem>freefall.FreeBSD.org</systemitem> on the <command>cvpasswd</command>
- command line, as described in this document,
- even when you are contacting
- <systemitem>cvsup-master.FreeBSD.org</systemitem>
- </para>
- </sect4>
</sect3>
</sect2>
</sect1>
@@ -897,14 +550,6 @@ cvspserver stream tcp nowait root /usr/b
what <emphasis>official tier-1 mirrors</emphasis>
are required to. All other official mirrors
can consider this a big <emphasis>should</emphasis>.</para>
- <note>
- <para>
- The following applies mainly to the FTP fileset,
- since a CVS repository should always be mirrored
- completely, and the web pages are a case of
- its own.
- </para>
- </note>
<para>
Tier-1 mirrors are required to:</para>
<itemizedlist>
More information about the svn-doc-all
mailing list