Building a mail application.. some advice appreciated

[Jeffrey Goldberg]

On Apr 25, 2007, at 8:43 AM, David Southwell wrote:

> Hi
>
> I am looking for some general advice and guidance for selecting  
> software
> components to fulfill a proof of concept test.
>
> I need a mail application with  features requiring that incoming  
> mails, which
> should comply with a predetermined format, be initially examined for
> compliance with that format

Presumably you are talking about the the format of the body of the  
message.  Or will the formated information all be in the headers?   
The tools you use will depend on that (and other things).

> Each sender (read user) has to be uniquely identified in the  
> database system
> and a log kept of every mail received.
>
> Sender verification requirements are high and, among other things,  
> the output
> from attachment processing must provide an input to the  
> verification system.

In order to help understand what you need it is important to break up  
the word "verification"  and by "sender".  In mail transport jargon  
"sender" means the address passed as the argument (an email address)  
to the MAIL FROM directive during SMTP.  Do you mean that or do you  
mean something else by sender.

By verification, do you just want to know whether it is a valid  
sender or do you want to authenticate the sender?  That is, do you  
want to have confidence that the sender is who they say they are?

When dealing with things like "verification" people like to break  
down the muddle into distinct tasks.  One is "identification".  For  
example, a username usually works to *identify* a particular user, so  
we know what user we are talking about.  "Authentication" is how we  
know that the person (or entity) wanting to use that identify really  
is who they say that are.  For example knowing a password is used to  
authenticate.  (Then there is "authorization") which I will leave  
aside for now).

For email there already are two good systems for authentication.  PGP  
and S/MIME.  Most toolkits for dealing with mail usually come with  
easy support for either of those.  For example, along with the many  
perl modules that exist for dealing with mail, there are modules for  
processing PGP or S/MIME signed messages.

> Mails that pass verification requirements are to be initially  
> processed by the
> receiving server and the results of verification transferred to a  
> mysql
> database.

Again, that depends on what the processing actually will be.  You  
will not do this within the MTA, but will pass (usually through an  
alias to a pipe, just like in mailman) the message to some program or  
script.  Whether it's written in perl, python, awk  etc is up to you,  
though there will already be nice packages in perl and python (and  
lots of other choices) for doing this kind of thing.  You will want a  
system that has nice integration with MySQL.  Again, all of the  
popular scripting languages do.

> Attachments  have to be extracted and passed for processing and   
> results
> stored  in a mysql db.

Again, you will want to use some system that has modules or libraries  
for dealing with email attachments.  Perl is what I'm most familiar  
with, but all the other ones will have such libraries/modules as well.

> Mails that do not comply with the verification requirements need to  
> be passed
> to another server for logging and processing.

When you say another "server" do you mean some other service to deal  
with these, some other mail server or some other host?

> The system has to be scaleable.
>
> I realise I have not given a lot of detailed information but here  
> is the rub I
> need to build, as quickly as possible, a proof of concept, using  
> readily
> available software components.

In a sense, any mailing list management system (that can use a mysql  
backend) already does what specify.  So do many other things that  
process mail (like customer relations systems, or bug/ticket  
reporting systems).

> The OS is freebsd (currently 6.1) running postfix.
>
> What components would you choose for this exercise? Be able to  
> build quickly
> and easily is the priority  for this stage and low server demand  
> would not be
> some important at this time.

I would really need to know what the problem is that you are trying  
to solve really is.  That way, we'll have a better understanding of  
whether email is the right solution, whether this has already been  
done, what kind of authentication really is needed, etc.  So tell us  
what you are trying to do with the system, and it will be easier to  
make meaningful recommendations.

Cheers,

-j