git: 6af7769111 - main - documentation/robots.txt: Disallow Past Releases Documentation

[Danilo G. Baio]

The branch main has been updated by dbaio:

URL: https://cgit.FreeBSD.org/doc/commit/?id=6af77691110d19019b389fad3a0700286fdf0ef1

commit 6af77691110d19019b389fad3a0700286fdf0ef1
Author:     Danilo G. Baio <dbaio at FreeBSD.org>
AuthorDate: 2021-08-08 14:12:03 +0000
Commit:     Danilo G. Baio <dbaio at FreeBSD.org>
CommitDate: 2021-08-13 22:51:16 +0000

    documentation/robots.txt: Disallow Past Releases Documentation
    
    It was noticed that Google/DDG sometimes are sending users to the past
    releases documentation. To avoid misinformation, disallow this archive
    from the search engines.
    
    This robots.txt only covers docs.FreeBSD.org and not www.FreeBSD.org, so
    also remove unnecessary items.
    
    Differential Revision: https://reviews.freebsd.org/D31459
---
 documentation/themes/beastie/layouts/robots.txt | 19 +------------------
 1 file changed, 1 insertion(+), 18 deletions(-)

diff --git a/documentation/themes/beastie/layouts/robots.txt b/documentation/themes/beastie/layouts/robots.txt
index b94bcd43cf..f59c629169 100644
--- a/documentation/themes/beastie/layouts/robots.txt
+++ b/documentation/themes/beastie/layouts/robots.txt
@@ -1,19 +1,2 @@
-# $FreeBSD$
-
 User-agent: *
-#Crawl-delay: 10
-Disallow: /cgi/confirm-code.cgi
-Disallow: /cgi/cvsweb.cgi
-Disallow: /cgi/dosendpr.cgi
-Disallow: /cgi/getmsg.cgi
-Disallow: /cgi/mailindex.cgi
-Disallow: /cgi/man.cgi?apropos=2&
-Disallow: /cgi/mid.cgi
-Disallow: /cgi/mirror.cgi
-Disallow: /cgi/missing_handler.cgi
-Disallow: /cgi/ports.cgi
-Disallow: /cgi/query-pr-summary.cgi
-Disallow: /cgi/query-pr.cgi
-Disallow: /cgi/search.cgi
-Disallow: /cgi/url.cgi
-Disallow: /statistic
+Disallow: /doc/