[OT] Re: What's the best possible email failover solution
cswiger at mac.com
Mon Jun 21 19:56:59 PDT 2004
Bill Moran wrote:
> Chuck Swiger <cswiger at mac.com> wrote:
>>[ I don't think that stuffing email into a database is a particularly good
>>idea since that means keeping large blobs of non-relational data floating
>>around, something that the filesystem can do a better job of handling... ]
[ ... ]
> During my research of the IMAP protocol, I determined that _the_best_
> way to store email for high-performance would be to put them in a
> database. This is because IMAP doesn't see email as a big blob of
> text like POP does. It sees the headers as one thing, and the
> different MIME parts of the email each as a seperate thing that can
> be fetched independently of the other MIME parts. This is a pretty
> good layout for a one -> many relationship in a database. Fact is,
> every current IMAP server that I'm aware of has to break emails
> apart on the fly in order to server IMAP.
There's nothing wrong with applying database concepts to email, and it sounds
like you want things which take advantage of database replication and
transaction management and so forth in order to gain reliability, so perhaps
you will find a DB better suited for your requirements than my comments above
I don't mind being wrong when the result works better for someone. However,
please remember that I know you are an optimist if you think I am a pessimist.
> Now, I could be wrong on this count, as I never wrote the mailserver,
> so my theory could ultimately be proven wrong, but I guess I just
> don't agree with the statement that SQL is a bad way to store email
> until someone has actually proven it.
My concern has less to do with the suitability of using a database to store
mail as it has to do with database transactions becoming a potential
bottleneck on the system as a whole.
I've spent a great deal of time in my day job dealing with dynamic websites,
which mostly means ones driven by content generated by a database. In my
experience, you want to provide static content as efficiently as possible, and
reserve database transactions for persisting changes to state and answering
The most relevant comparison is one involving a site where people can search
for images by keyword, which someone was also storing in the database. The
idea works fine under light to moderate load, but it turns out that keeping
just the "relational" part of the image data (name, keywords, etc) and a
filesystem reference, and generating a link using that path for Apache to
serve directly scales much better.
In the case of storing email in a DB, while you can break up a mail message
into headers plus seperate MIME components, are you really going to want to
decompose each and every mail message in a 3GB mail volume like that?
Although if you throw enough RAM at a DB so that the entire thing fits into
main memory, that can produce some spectacular results, and is almost doable
for this specific case.
Anyway, consider each time someone reads a message from the DB, you'd have to
do two or three database transactions per message, maybe more, compared with
read()ing or mmap()ing a single file in an IMAPD and doing strnstr()s for MIME
boundary seperators in C. Remember that hitting the DB involves multiprocess
IPC and adds a lot of latency compared to what a filesystem-based IMAP daemon
More information about the freebsd-questions