www/159652: PR is not visible to search engines

[Simon L. B. Nielsen]

The following reply was made to PR www/159652; it has been noted by GNATS.

From: "Simon L. B. Nielsen" <simon at FreeBSD.org>
To: Marcin Wisnicki <mwisnicki+freebsd at gmail.com>
Cc: freebsd-gnats-submit at FreeBSD.org
Subject: Re: www/159652: PR is not visible to search engines
Date: Sun, 21 Aug 2011 11:05:37 +0200

 On 10 Aug 2011, at 19:19, Marcin Wisnicki wrote:
 
 > Currently www.freebsd.org/robots.txt disallows access to /cgi/ and =
 that includes query-pr.cgi which makes bugs invisible to search engines =
 :(
 
 The problem is that query-cgi allows searches which are quiet heavy =
 resource wise (taking minutes to finish). If we just remove the =
 robots.txt from that area and the query cgi any reasonable amount of =
 hardware won't be enough. We even have extra protection which only =
 allows two query-pr's to run at once to avoid killing www.FreeBSD.org.
 
 I think it would be very nice if we could allow this, but it requires =
 someone to figure out a way to make query-pr.cgi not do searches when =
 indexes by robots. The simple, and also rather clean solution IMO, would =
 probably be to separate the viewing of a single PR, and searching for =
 PR's into different scripts - then we can allow allow robots to index =
 the real PR's without a problem. The problem for that is that somebody =
 has to do it and somebody else has to review before it's put into =
 production.
 
 --=20
 Simon L. B. Nielsen
 Hat: FreeBSD.org admins team