HAST and CARP

[Mikolaj Golub]

On Fri, 02 Jul 2010 16:59:30 +0900 hiroshi at soupacific.com wrote:

 h> On 7/2/2010 4:11 PM, Mikolaj Golub wrote:

 >>
 >> So you have:
 >>
 >> secondary localcnt:        1
 >> secondary remotecnt:        0
 >> primary localcnt:        1
 >> primary remotecnt:        0
 >>
 >> This is a split-brain condition as described on wiki: primary's localcnt is
 >> greater than secondary's remotecnt (primary [fw01A] was modified while fw01B
 >> wasn't watching) and secondary's localcnt is greater than primary's remotecnt
 >> (fw01B was modified while fw01A wasn't watching).

 h> So hasctl role secondary xxx does not change cnt values ?

Every node actually has only two values: localcnt and remotecnt. These values
are kept in disk metadata. So if a node behaves as a secondary, it reads these
values from the disk and stores in secondary_localcnt and
secondary_remotecnt. Other two values (primary_localcnt, primary_remotecnt)
are received from primary host. If I haven't overlooked something in the code
secondary does not modify localcnt and remotecnt in metadata. These counters
can be modified only when the node behaves as primary: on initialization
primary sets them to localcnt=1 and remotecnt=0, then if data are synchronized
between the nodes it sets the counters to the same values as on secondary. If
primary can't send data to secondary it increases localcnt.

So only primary can modify counters and if split-brain is detected that means
that secondary in past was primary for some time and another node was not
aware about this (or data was not synchronized).

 h> Scenario is this
 h> ServerA failed, then ServerB became MASTER.

 h> Only ServerA is started(say after fixed something) , both servers are
 h> connected,then ServerB starts,  BUT during failure of ServerA, ServerB
 h> was MASTER.
 h> ServerA was started before ServerB is started, thus ServerA should be
 h> MASTER!

 h> On this situation, CARP will set ServerA is MASTER and late comer
 h> ServerB is set as BACKUP by CARP.

 h> hastctl role secondary xxx set

 >> secondary localcnt:        1
 >> secondary remotecnt:        0
 >> primary localcnt:        1
 >> primary remotecnt:        0

 h> above values to NOT split-brain. It sounds more favoritabel way ????

I am not sure I understand what you mean here :-) 

First of all you can't modify counters manually. This is maintained by hast
internally.

When ServerB has become master and modified some data, ServerA, before setting
to primary again, should be set to secondary to synchronize all changes and
only after this be switched to primary, otherwise you will have split-brain
and should synchronize full storage recreating provider on secondary.

-- 
Mikolaj Golub