git: 67b05e8601 - main - block SEO bots from accessing cgi scripts

Go to: [ bottom of page ] [ top of archives ] [ this month ]

From: Wolfram Schneider <wosch_at_FreeBSD.org>
Date: Tue, 03 Oct 2023 12:26:20 UTC

The branch main has been updated by wosch:

URL: https://cgit.FreeBSD.org/doc/commit/?id=67b05e86012db7e1be22ab28203f8bc9f6a8148a

commit 67b05e86012db7e1be22ab28203f8bc9f6a8148a
Author:     Wolfram Schneider <wosch@FreeBSD.org>
AuthorDate: 2023-10-03 12:26:09 +0000
Commit:     Wolfram Schneider <wosch@FreeBSD.org>
CommitDate: 2023-10-03 12:26:09 +0000

    block SEO bots from accessing cgi scripts
---
 website/themes/beastie/layouts/robots.txt | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/website/themes/beastie/layouts/robots.txt b/website/themes/beastie/layouts/robots.txt
index b3bb9e880b..70cc4362be 100644
--- a/website/themes/beastie/layouts/robots.txt
+++ b/website/themes/beastie/layouts/robots.txt
@@ -11,3 +11,21 @@ Disallow: /cgi
 User-agent: CCBot
 Disallow: /cgi
 
+# SEO
+User-agent: SemrushBot
+Disallow: /cgi
+
+User-agent: MJ12bot
+Disallow: /cgi
+
+User-agent: BLEXBot
+Disallow: /cgi
+
+User-agent: dotbot
+Disallow: /cgi
+
+User-agent: SeekportBot
+Disallow: /cgi
+
+User-agent: AhrefsBot
+Disallow: /cgi