git: 82284de091 - main - remove non-standard Crawl-delay directive

Go to: [ bottom of page ] [ top of archives ] [ this month ]

From: Wolfram Schneider <wosch_at_FreeBSD.org>
Date: Sat, 02 Sep 2023 07:38:42 UTC

The branch main has been updated by wosch:

URL: https://cgit.FreeBSD.org/doc/commit/?id=82284de091cf1a05aa33538a49d6990ebbc043e3

commit 82284de091cf1a05aa33538a49d6990ebbc043e3
Author:     Wolfram Schneider <wosch@FreeBSD.org>
AuthorDate: 2023-09-02 07:31:36 +0000
Commit:     Wolfram Schneider <wosch@FreeBSD.org>
CommitDate: 2023-09-02 07:31:36 +0000

    remove non-standard Crawl-delay directive
---
 documentation/themes/beastie/layouts/robots.txt | 1 -
 website/themes/beastie/layouts/robots.txt       | 1 -
 2 files changed, 2 deletions(-)

diff --git a/documentation/themes/beastie/layouts/robots.txt b/documentation/themes/beastie/layouts/robots.txt
index fd409b7c52..2499651555 100644
--- a/documentation/themes/beastie/layouts/robots.txt
+++ b/documentation/themes/beastie/layouts/robots.txt
@@ -1,5 +1,4 @@
 User-agent: *
-#Crawl-delay: 10
 Disallow: /statistic
 Disallow: /doc/
 
diff --git a/website/themes/beastie/layouts/robots.txt b/website/themes/beastie/layouts/robots.txt
index 27f4152f84..8b6d6fb9b0 100644
--- a/website/themes/beastie/layouts/robots.txt
+++ b/website/themes/beastie/layouts/robots.txt
@@ -1,5 +1,4 @@
 User-agent: *
-#Crawl-delay: 10
 Disallow: /cgi/getmsg.cgi
 Disallow: /cgi/mailindex.cgi
 Disallow: /cgi/man.cgi?apropos=2&