svn commit: r42831 - head/en_US.ISO8859-1/htdocs
Glen Barber
gjb at FreeBSD.org
Fri Oct 4 18:57:43 UTC 2013
Author: gjb
Date: Fri Oct 4 18:57:43 2013
New Revision: 42831
URL: http://svnweb.freebsd.org/changeset/doc/42831
Log:
Explicitly list the files within cgi/, since not all crawlers will
honor the following syntax:
Allow: /cgi/foo
Disallow: /cgi/
Modified:
head/en_US.ISO8859-1/htdocs/robots.txt
Modified: head/en_US.ISO8859-1/htdocs/robots.txt
==============================================================================
--- head/en_US.ISO8859-1/htdocs/robots.txt Fri Oct 4 18:55:55 2013 (r42830)
+++ head/en_US.ISO8859-1/htdocs/robots.txt Fri Oct 4 18:57:43 2013 (r42831)
@@ -2,6 +2,21 @@
User-agent: *
#Crawl-delay: 10
-Disallow: /cgi/
+Disallow: /cgi/confirm-code.cgi
+Disallow: /cgi/cvsweb.cgi
+Disallow: /cgi/dosendpr.cgi
+Disallow: /cgi/getmsg.cgi
+Disallow: /cgi/mailindex.cgi
+Disallow: /cgi/man.cgi
+Disallow: /cgi/mid.cgi
+Disallow: /cgi/mirror.cgi
+Disallow: /cgi/missing_handler.cgi
+Disallow: /cgi/monthly.cgi
+Disallow: /cgi/pds.cgi
+Disallow: /cgi/ports.cgi
+Disallow: /cgi/query-pr-summary.cgi
+Disallow: /cgi/query-pr.cgi
+Disallow: /cgi/search.cgi
+Disallow: /cgi/url.cgi
Disallow: /statistic
Disallow: /sites/downloads.sourceforge.net/
More information about the svn-doc-all
mailing list