Production use of carp?

[John]

Hi Folks,

   Posting to -net & -fs to hopefully catch the right folks.
A similar posting to -current didn't seem to catch anyones
interest.  Please respond as approriate.

   I'm in the process of setting up HA/Failover ZFS server
systems using carp. I seem to be running into some issues that
may simply be misundersandings, or actual support issues. I'm
curious to hear what you think.

   First off, when using carp, one must use a unique vhid in
the configuration line for each system. If not, systems using
the same vhid, but different passwords will see a serious amount
of "jitteryness" and/or delay to their carp'd interface. This
means a unique set of vhid values would need to be assigned and
kept track of for every system put in place. Not something I
want to do. I've already run into this problem with another
group that was using carp on external interfaces to control
an HA nagios setup.

   Instead of running carp on the external interfaces as below:

ifconfig_cxgb0="inet 10.24.99.11 netmask 255.255.0.0"  # System 1 physical ip
ifconfig_cxgb0="inet 10.24.99.12 netmask 255.255.0.0"  # System 2 physcial ip
ifconfig_carp1="vhid 1 pass zfscarp1 advbase 1 advskew 100 10.24.99.13 netmask 255.255.0.0" # HA ip used by clients
  
   ... we instead connect a direct cross-over cable between the two systems
providing HA/Failover and use a private (backside) network:

ifconfig_cxgb1="inet 192.168.0.1 netmask 255.255.255.0"  # System 1 private ip
ifconfig_cxgb1="inet 192.168.0.2 netmask 255.255.255.0"  # System 2 private ip
ifconfig_carp1="vhid 1 pass zfscarp1 advbase 1 advskew 100 192.168.0.3 netmask 255.255.255.0"

   If system A is the MASTER, and I issue a 'ifconfig carp1 down'
command, system B becomes the MASTER as one would expect (using
scripts connected up through devd). So far, things are great. A
filesystem resource can be shifted to either A or B with no
impact on the clients.  Other scripts hooked up via devd monitor
the outgoing link and issue ifconfig carp1 up/down commands as
needed (for instance if the networking cable is unplugged on
head B).

   However, if system A is the MASTER, and system B is rebooted,
the carp interface on system A will flip/flop going down and
coming back up which is not what I want.

   This leads to my question, am I missing something simple about
using carp?  Should I implement my own control interface on the
private network and not use carp? What are other folks doing?

Thanks,
John