HAST: split-brain -- how to force one side to become primary?

[Freddie Cash]

According to the wiki, when a split-brain situation arises, I should be able
to stop hastd on one side, write changes to the /dev/hast/* providers on the
primary to increment the localcnt valye, and then bring up the secondary
hastd.  The locacnt/remotecnt values will be different, and everything will
start to re-sync.

However, this doesn't seem to work.  Or, maybe I'm not doing things right to
make it work.  Or maybe I've completely misunderstood how it all works.
 (Nah, that can never happen.  roll-eyes)  :)

/dev/hast/* is used to form a raidz1 vdev as part of pool "hapool".  There's
a single 1 GB zvol created, that is exported via iSCSI (net/istgt).  I can
mount the iSCSI disk on a Linux client, partition it, format it using XFS,
and write data to it.  Using only hast2 node as primary, I've written out 10
MB of new data, and verified that the data is there via "zfs list" on hast2,
and multiple mount/unmount cycles on the client.  Yet localcnt never
increments beyond 1 (remotecnt is 0).

Is there a way to forcibly increment localcnt on one node, so that bringing
up hastd on the other node will correctly come up as secondary, and start a
sync?

Or, do I have to manually re-do the HAST setup on one side?  Or zero out the
base/physical disk underneath HAST?

/etc/hast.conf (only listen line is different between nodes):
# Global section
control         /var/run/hastctl
listen          172.20.0.1
replication     memsync


# Resource section
resource disk01 {
        on hast1 {
                local   /dev/label/disk01
                remote  172.20.0.2
        }

        on hast2 {
                local   /dev/label/disk01
                remote  172.20.0.1
        }
}

resource disk02 {
        on hast1 {
                local   /dev/label/disk02
                remote  172.20.0.2
        }

        on hast2 {
                local   /dev/label/disk02
                remote  172.20.0.1
        }
}

resource disk03 {
        on hast1 {
                local   /dev/label/disk03
                remote  172.20.0.2
        }

        on hast2 {
                local   /dev/label/disk03
                remote  172.20.0.1
        }
}

resource disk04 {
        on hast1 {
                local   /dev/label/disk04
                remote  172.20.0.2
        }

        on hast2 {
                local   /dev/label/disk04
                remote  172.20.0.1
        }
}

hastctl dump on hast1:
resource: disk01
    datasize: 2147478528
    extentsize: 2097152
    keepdirty: 64
    localoff: 4608
    resuid: 1224151284752404553
    localcnt: 1
    remotecnt: 0
    prevrole: primary
resource: disk02
    datasize: 2147478528
    extentsize: 2097152
    keepdirty: 64
    localoff: 4608
    resuid: 10884849062207686761
    localcnt: 1
    remotecnt: 0
    prevrole: primary
resource: disk03
    datasize: 2147478528
    extentsize: 2097152
    keepdirty: 64
    localoff: 4608
    resuid: 14443609578994823508
    localcnt: 1
    remotecnt: 0
    prevrole: primary
resource: disk04
    datasize: 2147478528
    extentsize: 2097152
    keepdirty: 64
    localoff: 4608
    resuid: 1365498106518463540
    localcnt: 1
    remotecnt: 0
    prevrole: primary

hastctl dump on hast2:
resource: disk01
    datasize: 2147478528
    extentsize: 2097152
    keepdirty: 64
    localoff: 4608
    resuid: 1224151284752404553
    localcnt: 1
    remotecnt: 0
    prevrole: primary
resource: disk02
    datasize: 2147478528
    extentsize: 2097152
    keepdirty: 64
    localoff: 4608
    resuid: 10884849062207686761
    localcnt: 1
    remotecnt: 0
    prevrole: primary
resource: disk03
    datasize: 2147478528
    extentsize: 2097152
    keepdirty: 64
    localoff: 4608
    resuid: 14443609578994823508
    localcnt: 1
    remotecnt: 0
    prevrole: primary
resource: disk04
    datasize: 2147478528
    extentsize: 2097152
    keepdirty: 64
    localoff: 4608
    resuid: 1365498106518463540
    localcnt: 1
    remotecnt: 0
    prevrole: primary

-- 
Freddie Cash
fjwcash at gmail.com