Recommended search engine for web pages and maybe email?

[Philip Hallstrom]

Hi all -
 	Recently I've found myself searching the freebsd ports web site 
quite frequently as well as some other online documentation (php, mysql, 
postgresql, freebsd faq/handbook) and it always bothers me because I know 
I can mirror that stuff and search it locally and in general cut down on 
their load.

It's been a long time since I've setup any search engines/spiders to do 
this sort of thing.  In the past I've used htdig and mnogosearch.

I was hoping someone out there could tell me which one of those (or a 
third such as openfts?) I should install and get going to save me some 
time trying them all out.

I think the only feature I really care about is being able to limit the 
search to a particular collection (freebsd ports, php manual, etc.)

As an aside, I've got about 60mb (~ 5,000 messages) stored in pine's mbox 
format that I occasionally grep through, but would be nice if it was 
little more advanced...

I've also thought maybe I should just host it on an external server, and 
only allow my hosts and google's indexer to crawl it and just rely on 
google.

Suggestions?  recommendations?

Thanks!

-philip