fault tolerant web servers on freebsd

Paul Procacci pprocacci at datapipe.com
Wed Apr 7 09:04:57 UTC 2010


On 4/7/2010 2:37 AM, Matthew Seaman wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 07/04/2010 08:15:16, Joshua Boyd wrote:
>
>> On Wed, Apr 7, 2010 at 3:00 AM, Andriy Gapon<avg at icyb.net.ua>  wrote:
>>
>>
>>> on 07/04/2010 09:05 Aristedes Maniatis said the following:
>>>
>>>> Until we get to 'database' everything is HA and quite easy to build and
>>>> manage. Having a clustered database solution is expensive and beyond
>>>> most smallish budgets. mysql and postgresql don't have anything
>>>> available that is quite ready yet (IMO), so you'll need to be talking to
>>>> the bigger (expensive) players about their clustered offerings.
>>>>
>>>
>> Master-master circular replication in mySQL probably fits the bill.
>> Master-slave requires a slave to promote itself to master, which can get
>> tricky.
>>
> Although with master-master you need to be really careful to only use
> one instance read-write at any one time.  In theory you can design your
> DB schema and SQL to work correctly with multiple masters; in practice
> virtually no downloadable applications will work like this.
>
> Also remember that MySQL replication runs in a single thread on a single
> CPU core.  It's quite easy for a busy DB master with plenty of CPU cores
> to go so fast the replica can't actually keep up even if the replica
> uses exactly the same hardware.  Anyone who has a really good solution
> to this problem is going to make themselves rich beyond the dreams of
> avarice...
>

I'd like to chime in and provide some ideas that we use here for some of
our clients which works quite well.  In regards to master/slave, it's
quite trival to use a shared storage device like a 3par in combo' with
some heartbeat software.  You have a floating ip that ping pongs back
and forth between two machines depending on the error conditions that
you set.  As for the 3par itself, it has fiber to multiple switches so
losing that connection generally means a natural disaster taking it
offline.  Multipath (teaming/bonding) configurations take care of switch
problems.  I suppose this isn't really master/slave in the real sense,
rather two independant "masters" that can potentially read from the same
dataset, but does provide a very high level of redundancy without "data
duplication".  As for shared storage devices, personally I'd only use
fault tolerant devices and stay away from an MSA for example.

In regards to master/master I agree with the above whole heartedly.  We
have some clients that use it, but occasionally run into problems where
one of the "masters" falls so far behind another master, we're forced to
take it out of the pool.  In our case, a quick and simple ipfw rule
blocking new connections tells our loadbalancer to quit sending new
connection here until it's caught up, but it's a royal PITA, but really
can't think of any other possible solution to this.

In addition, we evaluated commercial middleware appliances offered by a
couple of companies and while they all worked, they were all terribly slow.

Also in regards to MysqlCluster, I had tested its throughput about a
year ago, thinking that would be the way to go for future master/master
deployments.  I don't remember the specific terminology so bear with me,
but as the number of "data stores" increased, the latency increased
along with it.  On top of this, I was only able to get around 300 qps.
This is abyssmal compared to a standalone machine.  Naturally this was
all on 8 core machines, tons of ram, and GigE interfaces.  Needless to
say, we scratched that idea real quick.  The cost to perfmance ratio
just wasn't worth it.  Maybe this has changed since I've last visited
this, though I'm uncertain.

Anyways, I hope this provides some useful insight.

~Paul

This message may contain confidential or privileged information.  If you are not the intended recipient, please advise us immediately and delete this message.  See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.


More information about the freebsd-stable mailing list