git: e09c06b9c2 - main - block CCBot (Common Crawl) from accessing cgi scripts

Go to: [ bottom of page ] [ top of archives ] [ this month ]

From: Wolfram Schneider <wosch_at_FreeBSD.org>
Date: Wed, 23 Aug 2023 06:17:52 UTC

The branch main has been updated by wosch:

URL: https://cgit.FreeBSD.org/doc/commit/?id=e09c06b9c2fa70e4884edae327d294c5efc3eb02

commit e09c06b9c2fa70e4884edae327d294c5efc3eb02
Author:     Wolfram Schneider <wosch@FreeBSD.org>
AuthorDate: 2023-08-23 06:17:36 +0000
Commit:     Wolfram Schneider <wosch@FreeBSD.org>
CommitDate: 2023-08-23 06:17:36 +0000

    block CCBot (Common Crawl) from accessing cgi scripts
---
 website/themes/beastie/layouts/robots.txt | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/website/themes/beastie/layouts/robots.txt b/website/themes/beastie/layouts/robots.txt
index 9bec7f76cf..1248457c75 100644
--- a/website/themes/beastie/layouts/robots.txt
+++ b/website/themes/beastie/layouts/robots.txt
@@ -14,3 +14,6 @@ Disallow: /statistic
 User-agent: GPTBot
 Disallow: /cgi
 
+User-agent: CCBot
+Disallow: /cgi
+