Getting CARP to broadcast on a different interface

Niklaas Baudet von Gersdorff stdin at
Thu Jun 9 17:54:45 UTC 2016

I can't believe it but I managed to do this:

Niklaas Baudet von Gersdorff [2016-06-08 18:30 +0200] :

> Then, I could use devd to assign the public failover IP (that I actually
> wanted to share with CARP on vtnet0) to the public interface vtnet0.
> CARP(4) provides an example on how to use carp status change events for
> additional scripting:
> --------------------8<--------------------
> Processing  of carp status change events can be set up by using the fol-
> lowing devd.conf rule:
>     notify 0 {
>         match "system"      "CARP";
>         match "subsystem"       "[0-9]+@[0-9a-z]+";
>         match "type"        "(MASTER|BACKUP)";
>         action "/root/ $subsystem $type";
>     };
> -------------------->8--------------------
> Depending von $type, could either
>     ifconfig vtnet0 <public-ip> alias
> or
>     ifconfig vtnet0 <public-ip> -alias
> I am not sure whether that works but I'll post my solution if I find
> one.

Obviously, the following set-up is very limited to a prober
configuration with CARP on a public network. However, it seems as if
supporting CARP in virtualised environments isn't possible because the
multicast packages would collide with those of other customers in the
same network. So the only solution is to use CARP within a private
network for some pseudo-CARP IP. Then, one can use the devd events that
are triggered if connection via the private network is lost, to assign
and remove the floating public IP that is used for failover.

So, let's see how this can be done. I have three hosts A, B, and C.
I could not manage to get CARP advertising over a virtual private
network of my provider that connects these hosts (CARP used the wrong
MAC addresses so the packets weren't translated correctly). But what
I managed to get going is CARP advertising over my tinc VPN tap0.

So if any host comes online and connects to the other hosts on the VPN,
tinc will assign a CARP vhid on tap0. This is
/usr/local/etc/tinc/host-up on host A:


ifconfig $INTERFACE vhid 1 advbase 01 advskew 000 pass <some-pasword> alias


On host B and C the file looks exactly the same except that advbase is
five seconds longer each (06 and 11 respectively). (While testing the
set-up I realised that latency is too high, so that advskew wouldn't

I use exactly the same devd.conf as quoted above. I put it under

Further, I did the following:

    sysctl net.inet.carp.preempt=1

which should be added to /etc/sysctl.conf.

/root/ is on every host and looks like the following:


     1  #!/bin/sh
     3  subsystem=$1
     4  type=$2
     5  vhid="1"
     6  c_if="tap0"             # CARP interface
     8  e_if="vtnet0"           # external interface
     9  IPv4="<floating-IPv4>"
    10  IPv4_net="/24"
    11  IPv6="<floating-IPv6>"
    12  IPv6_net="/64"
    14  log_tag=""
    16  remove_floating_ip() {
    17      if ifconfig $e_if | grep $IPv4 >/dev/null
    18      then
    19          ifconfig $e_if $IPv4$IPv4_net -alias && logger -t $log_tag "Removed $IPv4 on $e_if."
    20      fi
    21      if ifconfig $e_if | grep $IPv6 >/dev/null
    22      then
    23          ifconfig $e_if inet6 $IPv6$IPv6_net -alias && logger -t $log_tag "Removed $IPv6 on $e_if."
    24      fi
    25  }
    27  add_floating_ip() {
    28      ifconfig $e_if $IPv4$IPv4_net alias && logger -t $log_tag "Assigned $IPv4 on $e_if."
    29      ifconfig $e_if inet6 $IPv6$IPv6_net alias && logger -t $log_tag "Assigned $IPv6 on $e_if."
    30  }
    32  if [ "$subsystem" = "$vhid@$c_if" ]
    33  then
    34      case $type in 
    35          "MASTER")
    36              add_floating_ip
    37              ;;
    38          "BACKUP")
    39              remove_floating_ip
    40              ;;
    41      esac
    42  elif [ "$subsystem" = "tinc-down" ]
    43  then
    44      remove_floating_ip
    45  fi


Probably that's not the best way to do it and probably it could be
extended but my sh-skills are quite bad. This works though.

The script should be self-explanatory; the only ting that might be
unclear are lines 42-45. This is triggered by /usr/local/etc/tinc-down:


/root/ tinc-down


So, in case I stop tincd on purpose, the floating IP will be removed and
becomes free for the other machines. The second host will take over the
IP after some seconds.

Obviously, there are quite some drawbacks to this but it serves my
purpose. Since the machines are virtualised I don't expect them to down
by some hardware failure. But I do need failover in case I want to
upgrade one machines and need to reboot it.

In case I need to do some maintenance, I cut the machine in question
from the private network. For some seconds services won't be reachable
but after a short while host B will take over. I can upgrade host A and
restart it eventually. When host A runs again, it will connect to the
tinc VPN and while advertise CARP on that network. So, host A will
becomes MASTER again, host B will switch to BACKUP, and C will remain

Nonetheless, this is not a perfect solution. "High-availability" (these
quotes are on purpose) as achieved this way, depends on the connectivity
between the tinc nodes. E.g., if tinc malfunctions and doesn't shutdown
properly, the floating IP will stay assigned to the public interface,
the other client will loose connectivity to the node, and an additional
MASTER will rise. In addition, I had problems using a faster advbase
because of latency, I guess, thus there will be some downtime if any
MASTER goes down. In this case, the problem was that two hosts could not
decide who should be MASTER because there were some connectivity
problems between them on the VPN.

The crux is that CARP, as implemented in FreeBSD, cannot advertise on
unicast. If it could, I would have been able to assign it to the private
virtual interface that connects the virtual machines. However, CARP, as
implemented in OpenBSD, can do so using a feature called "carppeer".
There have been requests whether/when that will be brought to FreeBSD
[1-3] but the questions remained without responses. 


So, where can I post feature requests? :-) Without a way for unicast
CARP packets, FreeBSD is of much less value for virtualised environments
where multicast packets are an issue because they are blocked by
providers. So, setting up a failover set-up becomes a mess (such as the
one above).

Niklaas Baudet von Gersdorff [2016-06-08 21:23 +0200] :

> For that to work I must bind processes to non-local IP addresses. How do
> I do that?
> I found this
> with some recommendations to do so with ipfw. Can I do something similar
> with pf? Or is there even another solution for binding to non-local
> addresses?

No longer necessary. I use a pf rule that redirect any traffic arriving
on a specific port (this could also be any traffic arriving on the
pseudo-CARP IP) to the jail where the load-balancer is listening.

Anyway, I'm happy for any comments and further suggestions.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <>

More information about the freebsd-questions mailing list