svn commit: r41802 - head/en_US.ISO8859-1/htdocs

[Eitan Adler]

On 1 June 2013 00:59, Glen Barber <gjb at freebsd.org> wrote:
> On Fri, May 31, 2013 at 10:54:45PM +0000, Eitan Adler wrote:
>> Author: eadler
>> Date: Fri May 31 22:54:45 2013
>> New Revision: 41802
>> URL: http://svnweb.freebsd.org/changeset/doc/41802
>>
>> Log:
>>   There appear to be configuration errors causing an absurd amount of links to be generated.
>>   For now, block access to search engines until the problem is properly resolved.
>>
>>   While here, add $FreeBSD$
>>
>> Modified:
>>   head/en_US.ISO8859-1/htdocs/robots.txt
>>
>> Modified: head/en_US.ISO8859-1/htdocs/robots.txt
>> ==============================================================================
>> --- head/en_US.ISO8859-1/htdocs/robots.txt    Fri May 31 19:10:06 2013        (r41801)
>> +++ head/en_US.ISO8859-1/htdocs/robots.txt    Fri May 31 22:54:45 2013        (r41802)
>> @@ -1,4 +1,8 @@
>> +# $FreeBSD$
>> +
>>  User-agent: *
>>
>>  Disallow: /cgi/
>>  Disallow: /statistic
>> +Disallow: /publish.html/hu-hu/
>
> This is wrong.
>
> http://www.freebsd.org/index.html/omgwtfbbqhax

I was fixing a specific instance reported by automated tools run by a
search engine.  A more general fix needs to implemented by
clusteradm at .

>> +Disallow: /cgi/
>
> This already exists in the file...
...

Interesting, I am still getting reports of these URLs being indexed.
I'll look into that.


-- 
Eitan Adler
Source, Ports, Doc committer
Bugmeister, Ports Security teams