Free BSD in Googles's Index

[Kris Kennaway]

On Thu, Apr 17, 2003 at 07:21:13PM -0400, Bill Moran wrote:
> Ashley Gillard wrote:
> >Dear Free BSD Representative:
> >
> >Our crawl team has discovered that Google is currently blocked from
> >crawling your site by the robot.txt that is on your site. We would like
> >to index your site at http://bento.freebsd.org to provide you better
> >coverage and are interested in working with you to do this. 
> >
> >Google's mission is to deliver the best search experience on the
> >Internet by making the world's information universally accessible and
> >useful. To find out more about Google.com visit:
> >http://www.google.com/webmasters/.
> >
> >I look forward to hearing from you.
> 
> First off, bento is a ports build cluster and it's probably impractical
> to have google index it, as the information probably changes before it
> can even reach your index.

Correct, it's not appropriate for google to index bento.  It contains
tens of gigabytes of log files.

> If you still think it would be practical to alter the robots.txt file,
> it's probably best to contact Satoshi Asami (asami at freebsd.org) as he
> appears to maintain that site.

Nope, that's mine.

Kris
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20030417/aa284503/attachment.bin