svn commit: r44849 - head/en_US.ISO8859-1/articles/hubs

Gavin Atkinson gavin at
Sat May 17 02:50:17 UTC 2014

Author: gavin
Date: Sat May 17 02:50:17 2014
New Revision: 44849

  Start to bring this up-to-date:
   - Drop all references to providing CVSup mirrors
   - Drop all references to providing anonymous CVS mirrors
   - Drop all references to mirroring using CVSup, instead use rsync
   - Update the recommended arguments to rsync in the examples
   - Updatre size estimates for the individual parts
   - Change one reference to CVS to instead mention SVN
  Note that we are still not accepting applications for new mirrors
  of this style.


Modified: head/en_US.ISO8859-1/articles/hubs/article.xml
--- head/en_US.ISO8859-1/articles/hubs/article.xml	Fri May 16 19:28:26 2014	(r44848)
+++ head/en_US.ISO8859-1/articles/hubs/article.xml	Sat May 17 02:50:17 2014	(r44849)
@@ -57,7 +57,7 @@
         and degree of completeness you want to mirror, a huge
         amount of disk space may be consumed. Also keep in mind
         that <emphasis>official</emphasis> mirrors are probably required to be
-        complete. The CVS repository and the web pages should
+        complete. The web pages should
         always be mirrored completely. Also note that the
         numbers stated here are reflecting the current
         state (at &rel2.current;-RELEASE/&rel.current;-RELEASE). Further development and
@@ -67,10 +67,9 @@
         Here are some approximate figures:
-        <listitem><para>Full FTP Distribution: 1.1 TB</para></listitem>
-        <listitem><para>CVS repository: 5.4 GB</para></listitem>
-        <listitem><para>CTM deltas: 3.2 GB</para></listitem>
-        <listitem><para>Web pages: 463 MB</para></listitem>
+        <listitem><para>Full FTP Distribution: 1.4 TB</para></listitem>
+        <listitem><para>CTM deltas: 10 GB</para></listitem>
+        <listitem><para>Web pages: 1GB</para></listitem>
         The current disk usage of FTP Distribution can be found at
@@ -113,7 +112,7 @@
         For a moderately visited site that offers
-        <application>Rsync</application>, you might
+        <application>rsync</application>, you might
         consider a current CPU with around 800MHz - 1 GHz,
         and at least 512MB RAM. This is probably the
         minimum you want for an <emphasis>official</emphasis>
@@ -127,7 +126,7 @@
         You also want to consider a fast disk subsystem.
-        Operations on the CVS repository require a fast
+        Operations on the SVN repository require a fast
         disk subsystem (RAID is highly advised). A SCSI
         controller that has a cache of its own can also
         speed up things since most of these services incur a
@@ -181,17 +180,9 @@
             <listitem><para><package>ftp/twoftpd</package>: As above.</para></listitem>
             <listitem><para><package>ftp/vsftpd</package>: The <quote>very secure</quote> ftpd.</para></listitem>
-            <listitem>
-              <para><package>ftp/wu-ftpd</package>: The ftpd from Washington
-                University. It has become infamous, because of the huge
-                amount of security issues that have been found in it.
-                If you do choose to use this software be sure to
-                keep it up to date.
-              </para>
-            </listitem>
-          <para>FreeBSD's <application>ftpd</application>, <application>proftpd</application>,
-          <application>wu-ftpd</application> and maybe <application>ncftpd</application>
+          <para>FreeBSD's <application>ftpd</application>, <application>proftpd</application>
+          and maybe <application>ncftpd</application>
           are among the most commonly used FTPds.
           The others do not have a large userbase among mirror sites.  One
           thing to consider is that you may need flexibility in limiting
@@ -260,109 +251,6 @@
-      <sect3 xml:id="mirror-serv-cvsup">
-        <title>CVSup (desired for CVS repository)</title>
-        <para>
-          <application>CVSup</application> is a very efficient way of distributing files.
-          It works similar to <application>rsync</application>, but was specially designed for
-          use with CVS repositories. If you want to offer the
-          FreeBSD CVS repository, you really want to consider
-          offering it via <application>CVSup</application>. It is possible to offer
-          the CVS repository via <application>AnonCVS</application>, FTP,
-          <application>rsync</application> or HTTP, but
-          people would benefit much more from <application>CVSup</application> access.
-          <application>CVSup</application> was developed by &;.
-          It is a bit tricky to install on non-FreeBSD platforms,
-          since it is written in Modula-3 and therefore requires
-          a Modula-3 environment. &a.jdp; has built a
-          stripped down version of M3 that is sufficient to
-          run <application>CVSup</application>, and can be installed much easier.
-          See <link xlink:href="">Ezm3</link>
-          for details. Related ports are:</para>
-          <itemizedlist>
-            <listitem>
-              <para><package>net/cvsup</package>: The native CVSup port (client and server)
-                which requires <package>lang/ezm3</package> now.</para>
-            </listitem>
-            <listitem>
-              <para><package>net/cvsup-mirror</package>: The CVSup mirror kit, which requires
-                 <package>net/cvsup-without-gui</package>, and configures it mirror-ready. Some
-                 site administrators may want a different setup though.
-              </para>
-            </listitem>
-          </itemizedlist>
-	<para>There are a few more like
-          <package>net/cvsup-without-gui</package> you might want to have
-          a look at. If you prefer a static binary package, take a look
-          <link xlink:href="">here</link>.
-          This page still refers to the S1G bug that was present
-          in <application>CVSup</application>. Maybe
-          John will set up a generic download-site to get
-          static binaries for various platforms.
-        </para>
-        <para>
-          It is possible to use <application>CVSup</application> to offer
-          any kind of fileset, not just CVS repositories,
-          but configuration can be complex.
-          <application>CVSup</application> is known to eat some CPU on both the server and the
-          client, since it needs to compare lots of files.
-        </para>
-      </sect3>
-      <sect3 xml:id="mirror-anoncvs">
-        <title>AnonCVS (optional for CVS repository)</title>
-        <para>
-          If you have the CVS repository, you may want to offer
-          anonymous CVS access. A short warning first:
-          There is not much demand for it,
-          it requires some experience, and you need to know
-          what you are doing.
-        </para>
-        <para>
-          Generally there are two ways
-          to access a CVS repository remotely: via
-          <emphasis>pserver</emphasis> or via <command>ssh</command>
-          (we do not consider <command>rsh</command>).
-          For anonymous access, <emphasis>pserver</emphasis> is
-          very well suited, but some still offer <command>ssh</command>
-          access as well. There is a custom crafted
-          <link xlink:href="">wrapper</link>
-          in the CVS repository, to be used as a login-shell for the
-          anonymous ssh account. It does a chroot, and therefore
-          requires the CVS repository to be available under the
-          anonymous user's home-directory.   This may not be possible
-          for all sites. If you just offer <emphasis>pserver</emphasis>
-          this restriction does not apply, but you may run with
-          more security risks. You do not need to install any special
-          software, since &man.cvs.1; comes with
-          FreeBSD. You need to enable access via <command>inetd</command>,
-          so add an entry into your <filename>/etc/inetd.conf</filename>
-          like this:</para>
-          <programlisting>
-cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --allow-root=/home/ncvs pserver
-          </programlisting>
-          <para>See the manpage for details of the options. Also see the CVS <emphasis>info</emphasis>
-          page about additional ways to make sure access is read-only.
-          It is advised that you create an unprivileged account,
-          preferably called <systemitem class="username">anoncvs</systemitem>.
-          Also you need to create a file <filename>passwd</filename>
-          in your <filename>/home/ncvs/CVSROOT</filename> and assign a
-          CVS password (empty or <literal>anoncvs</literal>) to that user.
-          The directory <filename>/anoncvstmp</filename> is a special
-          purpose memory based file system. It is not required but
-          advised since &man.cvs.1; creates a shadow directory
-          structure in your <filename>/tmp</filename> which is
-          not used after the operation but slows things
-          dramatically if real disk operations are required.
-          Here is an excerpt from <filename>/etc/fstab</filename>,
-          how to set up such a MFS:</para>
-          <programlisting>
-/dev/da0s1b /anoncvstmp mfs rw,-s=786432,-b=4096,-f=512,-i=560,-c=3,-m=0,nosuid 0 0
-          </programlisting>
-          <para>This is (of course) tuned a lot, and was suggested by &;.
-        </para>
-      </sect3>
   <sect1 xml:id="mirror-howto">
@@ -374,8 +262,8 @@ cvspserver stream tcp nowait root /usr/b
       the various parts of FreeBSD, what tools to use,
       and where to mirror from.
-    <sect2 xml:id="mirror-ftp">
-    <title>FTP</title>
+    <sect2 xml:id="mirror-ftp-rsync">
+    <title>Mirroring the FTP site</title>
         The FTP area is the largest amount of data that
         needs to be mirrored. It includes the <emphasis>distribution
@@ -388,39 +276,8 @@ cvspserver stream tcp nowait root /usr/b
         for various FreeBSD versions,
         and various architectures.
-      <sect3 xml:id="mirror-ftp-ftp">
-        <title>With FTP mirror</title>
-          You can use a <application>FTP mirror</application>
-          program to get the files. Some of the most commonly used are:</para>
-          <itemizedlist>
-            <listitem><para><package>ftp/mirror</package></para></listitem>
-            <listitem><para><package>ftp/ftpmirror</package></para></listitem>
-            <listitem><para><package>ftp/emirror</package></para></listitem>
-            <listitem><para><package>ftp/spegla</package></para></listitem>
-            <listitem><para><package>ftp/omi</package></para></listitem>
-            <listitem><para><package>ftp/wget</package></para></listitem>
-          </itemizedlist>
-          <para><package>ftp/mirror</package> was very popular, but seemed
-          to have some drawbacks, as it is written in &man.perl.1;,
-          and had real problems with mirroring large
-          directories like a FreeBSD site. There are rumors that
-          the current version has fixed this by allowing
-          a different algorithm for comparing
-          the directory structure to be specified.
-        </para>
-        <para>
-          In general FTP is not really good for mirroring.  It transfers
-          the whole file if it has changed, and does
-          not create a single data stream which would benefit from
-          a large TCP congestion window.
-        </para>
-      </sect3>
-      <sect3 xml:id="mirror-ftp-rsync">
-        <title>With rsync</title>
-        <para>
-          A better way to mirror the FTP area is <application>rsync</application>.
+          The best way to mirror the FTP area is <application>rsync</application>.
           You can install the port <package>net/rsync</package> and then use
           rsync to sync with your upstream host.
           <application>rsync</application> is already mentioned
@@ -440,7 +297,7 @@ cvspserver stream tcp nowait root /usr/b
           <para>A command line to mirror FreeBSD might look like:</para>
-          <screen>&prompt.user; <userinput>rsync -vaz --delete /pub/FreeBSD/</userinput>
+          <screen>&prompt.user; <userinput>rsync -vaHz --delete rsync:// /pub/FreeBSD/</userinput>
           <para>Consult the documentation for <application>rsync</application>,
           which is also available at
@@ -453,188 +310,15 @@ cvspserver stream tcp nowait root /usr/b
           want to set up a script framework that calls such a command
           via &man.cron.8;.
-      </sect3>
-      <sect3 xml:id="mirror-ftp-cvsup">
-        <title>With CVSup</title>
-        <para>
-          A few sites, including the one-and-only <systemitem class="fqdomainname"></systemitem>
-          even offer <application>CVSup</application> to mirror the contents of
-          the FTP space. You need to install a <application>CVSup</application>
-          client, preferably from the port <package>net/cvsup</package>.
-          (Also reread <xref linkend="mirror-serv-cvsup"/>.)
-          A sample <filename>supfile</filename> suitable for <systemitem class="fqdomainname"></systemitem>
-          looks like this:</para>
-          <programlisting>
-          #
-          # FreeBSD archive supfile from master server
-          #
-          *default
-          *default base=/usr
-          *default prefix=/pub
-          #*default release=all
-          *default delete use-rel-suffix
-          *default umask=002
-          # If your network link is a T1 or faster, comment out the following line.
-          #*default compress
-          FreeBSD-archive release=all preserve
-          </programlisting>
-          <para>It seems <application>CVSup</application> would be the best
-          way to mirror the archive in terms of efficiency, but
-          it is only available from few sites.</para>
-          <note xml:id="mirror-cvsup-s-option">
-            <para>
-              Please have look at the <application>CVSup</application> documentation
-              like &man.cvsup.1; and consider using the <option>-s</option>
-	      option. This reduces I/O operations by assuming the
-	      recorded information about each file is correct.</para>
-          </note>
-      </sect3>
-    </sect2>
-    <sect2 xml:id="mirror-cvs">
-      <title>Mirroring the CVS repository</title>
-      <para>There are various ways to mirror the CVS repository.
-        <application>CVSup</application> is the most common method.</para>
-      <sect3 xml:id="mirror-cvs-cvsup">
-        <title>Using CVSup</title>
-        <para>
-          <application>CVSup</application> is described in some
-          detail in <xref linkend="mirror-serv-cvsup"/> and <xref linkend="mirror-ftp-cvsup"/>.
-        </para>
-        <para>It is very easy to setup a
-           <application>CVSup</application> mirror. Installing
-           <package>net/cvsup-mirror</package> will
-           make sure all of the needed programs are installed and then
-           gather all the needed information to configure the mirror.</para>
-        <note>
-          <para>
-            Please do not forget to consider the hint
-            mentioned in <link linkend="mirror-cvsup-s-option">this note</link>
-            above.
-          </para>
-        </note>
-      </sect3>
-      <sect3 xml:id="mirror-cvs-other">
-        <title>Using other methods</title>
-        <para>
-          Using other methods than <application>CVSup</application> is
-          generally not recommended. We describe them in short here
-          anyway. Since most sites offer the CVS repository as
-          part of the FTP fileset under the path
-          <filename>/pub/FreeBSD/development/FreeBSD-CVS</filename>,
-          the following methods could be used.</para>
-          <itemizedlist>
-            <listitem><para><application>FTP</application></para></listitem>
-            <listitem><para><application>Rsync</application></para></listitem>
-            <listitem><para><application>HTTP</application></para></listitem>
-          </itemizedlist>
-          <important>
-            <para>AnonCVS cannot be used to mirror the CVS repository
-              since CVS does not allow you to access the repository
-              itself, only checked out versions of the modules.</para>
-          </important>
-      </sect3>
     <sect2 xml:id="mirror-www">
       <title>Mirroring the WWW pages</title>
-        The best way is to check out the <emphasis>www</emphasis>
-        distribution from CVS. If you have a local mirror of the
-        CVS repository, it is as easy as:</para>
-        <screen>&prompt.user; <userinput>cvs -d /home/ncvs co www</userinput></screen>
-      <para>and a <emphasis>cronjob</emphasis>, that calls <command>cvs up -d -P</command>
-        on a regular basis, maybe just after your repository was updated.
-        Of course, the files need to remain in a directory available
-        for public WWW access. The installation and configuration of a
-        web server is not discussed here.
-      </para>
-      <para>
-        If you do not have a local repository, you can use
-        <application>CVSup</application> to maintain an <quote>up to date copy</quote>
-        of the www pages. A sample supfile can be found in
-        <filename>/usr/share/examples/cvsup/www-supfile</filename> and
-        could look like this:</para>
-        <programlisting>
-        #
-        # WWW module supfile for FreeBSD
-        #
-        *default
-        *default base=/usr
-        *default prefix=/usr/local
-        *default release=cvs tag=.
-        *default delete use-rel-suffix
-        # If your network link is a T1 or faster, comment out the following line.
-        *default compress
-        # This collection retrieves the www/ tree of the FreeBSD repository
-        www
-        </programlisting>
-     <para>
-       Using <package>ftp/wget</package> or other web-mirror tools is
-       not recommended.
-     </para>
-     <sect3 xml:id="mirror-www-doc">
-       <title>Mirroring the FreeBSD documentation</title>
-       <para>
-         Since the documentation is referenced a lot from the
-         web pages, it is recommended that you mirror the
-         FreeBSD documentation as well. However, this is not
-         as trivial as the www-pages alone.
-       </para>
-       <para>
-         First of all, you should get the doc sources,
-         again preferably via <application>CVSup</application>.
-         Here is a corresponding sample supfile:</para>
-         <programlisting>
-         #
-         # FreeBSD documentation supfile
-         #
-         *default
-         *default base=/usr
-         *default prefix=/usr/share
-         *default release=cvs tag=.
-         *default delete use-rel-suffix
-         # If your network link is a T1 or faster, comment out the following line.
-         #*default compress
-         # This will retrieve the entire doc branch of the FreeBSD repository.
-         # This includes the handbook, FAQ, and translations thereof.
-         doc-all
-         </programlisting>
-       <para>
-         Then you need to install a couple of ports.
-         You are lucky, there is a meta-port:
-         <package>textproc/docproj</package> to do the work
-         for you. You need to set up some
-         environment variables, like
-         <literal>SGML_CATALOG_FILES</literal>.
-         Also have a look at your <filename>/etc/make.conf</filename>
-         (copy <filename>/usr/share/examples/etc/make.conf</filename> if
-         you do not have one), and look at the
-         <literal>DOC_LANG</literal> variable.
-         Now you are probably ready to run <command>make</command>
-         in your doc directory (<filename>/usr/share/doc</filename>
-         by default) and build the documentation.
-         Again you need to make it accessible for your web server
-         and make sure the links point to the right location.</para>
-         <important>
-           <para>
-             The building of the documentation, as well as lots
-             of side issues, is documented itself in the
-	     <link xlink:href="&url.books.fdp-primer;">&os; Documentation
-             Project Primer</link>.
-             Please read this piece of documentation, especially if you
-             have problems building the documentation.
-           </para>
-         </important>
-     </sect3>
+	The FreeBSD website should only be mirrored via
+	<application>rsync</application>.</para>
+      <para>A command line to mirror the FreeBSD web site might look like:</para>
+      <screen>&prompt.user; <userinput>rsync -vaHz --delete rsync:// /usr/local/www/</userinput>
+      </screen>
    <sect2 xml:id="mirror-how-often">
      <title>How often should I mirror?</title>
@@ -683,7 +367,6 @@ cvspserver stream tcp nowait root /usr/b
        Here are some recommended schedules:</para>
          <listitem><para>FTP fileset: daily</para></listitem>
-         <listitem><para>CVS repository: hourly</para></listitem>
          <listitem><para>WWW pages: daily</para></listitem>
@@ -708,7 +391,7 @@ cvspserver stream tcp nowait root /usr/b
         <emphasis>N</emphasis> is a number,
         telling that the host would be the <emphasis>Nth</emphasis>
         mirror in that country.
-        (Same applies to <systemitem></systemitem>,
+        (Same applies to 
         <systemitem></systemitem>, etc.)
         There are mirrors with no <emphasis>CC</emphasis> part.
         These are the mirror sites that are very well connected and
@@ -812,9 +495,7 @@ cvspserver stream tcp nowait root /usr/b
-          There are three master sites for the FTP fileset and
-          one for the CVS repository (the web pages and docs are
-          obtained from CVS, so there is no need for master).
+          There is one master site for the FTP fileset.
         <sect4 xml:id="mirror-where-master-ftp">
@@ -823,11 +504,9 @@ cvspserver stream tcp nowait root /usr/b
             <systemitem></systemitem> provides
-            <application>rsync</application> and <application>CVSup</application>
+            <application>rsync</application>
             access, in addition to FTP.
-            Refer to <xref linkend="mirror-ftp-rsync"/> and
-            <xref linkend="mirror-ftp-cvsup"/> how to access
-            via these protocols.
+            Refer to <xref linkend="mirror-ftp-rsync"/>.
             Mirrors are also encouraged to allow <application>rsync</application>
@@ -835,32 +514,6 @@ cvspserver stream tcp nowait root /usr/b
-        <sect4 xml:id="mirror-where-master-cvsup">
-          <title></title>
-          <para>
-            This is the master site for the CVS repository.
-          </para>
-          <para>
-            <systemitem></systemitem> provides
-            <application>CVSup</application> access only.
-            See <xref linkend="mirror-cvs-cvsup"/> for details.
-          </para>
-          <para>
-            To get access, you need to contact the &a.cvsup-master;.
-            Make sure you read the
-            <link xlink:href="">FreeBSD CVSup Access Policy</link>
-            first!
-          </para>
-          <para>
-            Set up the required authentication by following
-            <link xlink:href="">these
-            instructions</link>. Make sure you specify the server as
-            <systemitem></systemitem> on the <command>cvpasswd</command>
-            command line, as described in this document,
-            even when you are contacting
-            <systemitem></systemitem>
-          </para>
-        </sect4>
@@ -897,14 +550,6 @@ cvspserver stream tcp nowait root /usr/b
         what <emphasis>official tier-1 mirrors</emphasis>
         are required to. All other official mirrors
         can consider this a big <emphasis>should</emphasis>.</para>
-        <note>
-          <para>
-            The following applies mainly to the FTP fileset,
-            since a CVS repository should always be mirrored
-            completely, and the web pages are a case of
-            its own.
-          </para>
-        </note>
         Tier-1 mirrors are required to:</para>

More information about the svn-doc-all mailing list