Free BSD in Googles's Index
kris at obsecurity.org
Thu Apr 17 17:56:55 PDT 2003
On Thu, Apr 17, 2003 at 07:21:13PM -0400, Bill Moran wrote:
> Ashley Gillard wrote:
> >Dear Free BSD Representative:
> >Our crawl team has discovered that Google is currently blocked from
> >crawling your site by the robot.txt that is on your site. We would like
> >to index your site at http://bento.freebsd.org to provide you better
> >coverage and are interested in working with you to do this.
> >Google's mission is to deliver the best search experience on the
> >Internet by making the world's information universally accessible and
> >useful. To find out more about Google.com visit:
> >I look forward to hearing from you.
> First off, bento is a ports build cluster and it's probably impractical
> to have google index it, as the information probably changes before it
> can even reach your index.
Correct, it's not appropriate for google to index bento. It contains
tens of gigabytes of log files.
> If you still think it would be practical to alter the robots.txt file,
> it's probably best to contact Satoshi Asami (asami at freebsd.org) as he
> appears to maintain that site.
Nope, that's mine.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20030417/aa284503/attachment.bin
More information about the freebsd-questions