[OT] Re: What's the best possible email failover solution

Chuck Swiger cswiger at mac.com
Mon Jun 21 19:56:59 PDT 2004

Bill Moran wrote:
> Chuck Swiger <cswiger at mac.com> wrote:
>>[ I don't think that stuffing email into a database is a particularly good 
>>idea since that means keeping large blobs of non-relational data floating 
>>around, something that the filesystem can do a better job of handling... ]
[ ... ]
> During my research of the IMAP protocol, I determined that _the_best_
> way to store email for high-performance would be to put them in a
> database.  This is because IMAP doesn't see email as a big blob of
> text like POP does.  It sees the headers as one thing, and the
> different MIME parts of the email each as a seperate thing that can
> be fetched independently of the other MIME parts.  This is a pretty
> good layout for a one -> many relationship in a database.  Fact is,
> every current IMAP server that I'm aware of has to break emails
> apart on the fly in order to server IMAP.

There's nothing wrong with applying database concepts to email, and it sounds 
like you want things which take advantage of database replication and 
transaction management and so forth in order to gain reliability, so perhaps 
you will find a DB better suited for your requirements than my comments above 

I don't mind being wrong when the result works better for someone.  However, 
please remember that I know you are an optimist if you think I am a pessimist.


> Now, I could be wrong on this count, as I never wrote the mailserver,
> so my theory could ultimately be proven wrong, but I guess I just
> don't agree with the statement that SQL is a bad way to store email
> until someone has actually proven it.

My concern has less to do with the suitability of using a database to store 
mail as it has to do with database transactions becoming a potential 
bottleneck on the system as a whole.

I've spent a great deal of time in my day job dealing with dynamic websites, 
which mostly means ones driven by content generated by a database.  In my 
experience, you want to provide static content as efficiently as possible, and 
reserve database transactions for persisting changes to state and answering 
relational queries.

The most relevant comparison is one involving a site where people can search 
for images by keyword, which someone was also storing in the database.  The 
idea works fine under light to moderate load, but it turns out that keeping 
just the "relational" part of the image data (name, keywords, etc) and a 
filesystem reference, and generating a link using that path for Apache to 
serve directly scales much better.


In the case of storing email in a DB, while you can break up a mail message 
into headers plus seperate MIME components, are you really going to want to 
decompose each and every mail message in a 3GB mail volume like that? 
Although if you throw enough RAM at a DB so that the entire thing fits into 
main memory, that can produce some spectacular results, and is almost doable 
for this specific case.

Anyway, consider each time someone reads a message from the DB, you'd have to 
do two or three database transactions per message, maybe more, compared with 
read()ing or mmap()ing a single file in an IMAPD and doing strnstr()s for MIME 
boundary seperators in C.  Remember that hitting the DB involves multiprocess 
IPC and adds a lot of latency compared to what a filesystem-based IMAP daemon 


More information about the freebsd-questions mailing list