From sebster at sebster.com Sat Apr 4 01:47:35 2009 From: sebster at sebster.com (Sebastiaan van Erk) Date: Sat Apr 4 01:47:43 2009 Subject: CARP and NIC-teaming on ESXi Message-ID: <49D71864.3010403@sebster.com> Hi all, I have a problem using CARP for IP failover on a FreeBSD 7.1-amd64 virtual machine on ESXi 3.5. The problem can be reproduced using a single FreeBSD virtual mmachine. When I put the FreeBSD VM on a virtual switch on ESXi with one physical network card associated to it, CARP works fine, the carp interface becomes MASTER. However, as soon as I associate 2 physical network cards with the virtual switch so that I have NIC failover, the FreeBSD machine continuously becomes the BACKUP even though there are no other machines with the same vhid. In the logs I get this message repeated ad infinitum: carp0: MASTER -> BACKUP (more frequent advertisement received) arp_rtrequest: bad gateway 192.168.1.1 (!AF_LINK) carp0: MASTER -> BACKUP (more frequent advertisement received) arp_rtrequest: bad gateway 192.168.1.1 (!AF_LINK) When I ping the shared IP addres 192.168.1.1 it replies VERY infrequently (a few times ever 50000 pings). When I do a tcpdump to see the carp advertisements then when it is working (only one network card), I see something like this (with carp advertisements approximately every second): 10:08:11.241213 00:00:5e:00:01:4a > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 28744, offset 0, flags [DF], proto VRRP (112), length 56) 192.168.1.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 74, prio 100, authtype none, intvl 1s, length 36, addrs(7): 156.231.51.105,143.57.8.21,37.96.2.205,99.70.81.166,146.170.184.206,103.60.18.123,240.32.224.52 10:08:12.651052 00:00:5e:00:01:4a > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 28757, offset 0, flags [DF], proto VRRP (112), length 56) 192.168.1.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 74, prio 100, authtype none, intvl 1s, length 36, addrs(7): 156.231.51.105,143.57.8.22,34.65.134.219,132.160.185.229,103.202.156.249,174.25.227.190,231.95.30.57 10:08:14.061257 00:00:5e:00:01:4a > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 28769, offset 0, flags [DF], proto VRRP (112), length 56) 192.168.1.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 74, prio 100, authtype none, intvl 1s, length 36, addrs(7): 156.231.51.105,143.57.8.23,91.0.105.24,240.239.51.204,21.9.216.6,232.26.58.127,73.8.235.226 However, when I do the NIC teaming on ESXi, I constantly see 2 packets arrive at the same time and I get the 'more frequent advertisement received' message in the logs: 10:10:47.527982 00:00:5e:00:01:4a > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 30136, offset 0, flags [DF], proto VRRP (112), length 56) 192.168.1.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 74, prio 100, authtype none, intvl 1s, length 36, addrs(7): 156.231.51.105,143.57.8.129,248.175.196.213,178.47.150.234,200.203.153.156,219.129.15.78,19.136.6.207 10:10:47.529163 00:00:5e:00:01:4a > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: (tos 0x10, ttl 255, id 30136, offset 0, flags [DF], proto VRRP (112), length 56) 192.168.1.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 74, prio 100, authtype none, intvl 1s, length 36, addrs(7): 156.231.51.105,143.57.8.129,248.175.196.213,178.47.150.234,200.203.153.156,219.129.15.78,19.136.6.207 Does anybody know what could be going on, and whether it's possible to get this working? Thanks in advance, Sebastiaan -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3328 bytes Desc: S/MIME Cryptographic Signature Url : http://lists.freebsd.org/pipermail/freebsd-cluster/attachments/20090404/97a9793a/smime.bin From lavalamp at spiritual-machines.org Mon Apr 6 11:13:40 2009 From: lavalamp at spiritual-machines.org (Brian A. Seklecki) Date: Mon Apr 6 11:13:47 2009 Subject: CARP and NIC-teaming on ESXi In-Reply-To: <49D71864.3010403@sebster.com> References: <49D71864.3010403@sebster.com> Message-ID: <1239040711.16390.2195.camel@soundwave.ws.pitbpa0.priv.collaborativefusion.com> On Sat, 2009-04-04 at 10:20 +0200, Sebastiaan van Erk wrote: > I see something like this (with carp > advertisements approximately every second): > > 10:08:11.241213 00:00:5e:00:01:4a There are a handful of settings buried deep in ESXi related to virtual switch promiscuous mode, multicast, and MAC spoofing (transmitting from the VRRP/HSRP/CARP address). You have to configure them in about 3-4 places. ~BAS From sebster at sebster.com Mon Apr 6 14:07:35 2009 From: sebster at sebster.com (Sebastiaan van Erk) Date: Mon Apr 6 14:07:42 2009 Subject: CARP and NIC-teaming on ESXi In-Reply-To: <1239040711.16390.2195.camel@soundwave.ws.pitbpa0.priv.collaborativefusion.com> References: <49D71864.3010403@sebster.com> <1239040711.16390.2195.camel@soundwave.ws.pitbpa0.priv.collaborativefusion.com> Message-ID: <49DA6F13.8000908@sebster.com> Hi, Thanks for the reply. I had already found the promiscuous mode of the virtual switch that the boxes are on, otherwise CARP does not work at all. CARP itself works (I have it set up with 2 real ESXi machines each with a virtual FreeBSD machine which is doing CARP, so that if either physical machine fails, the CARP kicks in). The only problem is that together with NIC teaming on ESXi it breaks; even when I just set the second NIC to standby mode. Do you have any idea where I can find the settings and which ones to tune? Regards, Sebastiaan Brian A. Seklecki wrote: > On Sat, 2009-04-04 at 10:20 +0200, Sebastiaan van Erk wrote: >> I see something like this (with carp >> advertisements approximately every second): >> >> 10:08:11.241213 00:00:5e:00:01:4a > > There are a handful of settings buried deep in ESXi related to virtual > switch promiscuous mode, multicast, and MAC spoofing (transmitting from > the VRRP/HSRP/CARP address). You have to configure them in about 3-4 > places. ~BAS > > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3328 bytes Desc: S/MIME Cryptographic Signature Url : http://lists.freebsd.org/pipermail/freebsd-cluster/attachments/20090406/ce536997/smime.bin From sebster at sebster.com Wed Apr 8 03:29:45 2009 From: sebster at sebster.com (Sebastiaan van Erk) Date: Wed Apr 8 03:29:52 2009 Subject: CARP, openvpn in bridged mode, and ping Message-ID: <49DC7C96.2050203@sebster.com> Hi, I have the following setup: two servers with a virtual LAN IP address shared with CARP (the hosts are 10.0.80.77 and 10.0.80.76 and the virtual IP address is 10.0.80.1). When I ping the VIP from any host on the LAN, it works fine. Next I have some openvpn clients (both 10.0.80.77 and 10.0.80.76 have openvpn servers on their external IPs). The client IPs are on the LAN using a bridge and are 10.0.80.150 (linux client) and 10.0.80.6 (freebsd client). From linux I can ping the VIP (10.0.80.1) just fine, but when I do arping I see (with tcpdump) that the the ARP requests are received by the carp master on the tap0 device, but that it does not reply. From a FreeBSD VPN client I cannot ping the VIP (10.0.80.1), because it does the ARP requests indefinitely and gets no answer. Both machines ping to the other hosts on the LAN just fine (e.g., all of them can ping 10.0.80.77 just fine). Is there any way to get ARP to work (and thereby, ping to work) in this configuration? Regards, Sebastiaan PS: the relevant ifconfig info is: 10.0.80.77 (carp master and vpn server): em1: flags=8943 metric 0 mtu 1500 options=98 ether 00:0c:29:61:2a:55 inet 10.0.80.77 netmask 0xffffff00 broadcast 10.0.80.255 media: Ethernet autoselect (1000baseTX ) status: active bridge0: flags=8843 metric 0 mtu 1500 ether 12:d8:09:8d:1b:88 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap0 flags=143 ifmaxaddr 0 port 9 priority 128 path cost 2000000 member: em1 flags=143 ifmaxaddr 0 port 2 priority 128 path cost 20000 carp1: flags=49 metric 0 mtu 1500 inet 10.0.80.1 netmask 0xffffff00 carp: MASTER vhid 174 advbase 1 advskew 0 tap0: flags=8943 metric 0 mtu 1500 ether 00:bd:c0:02:00:00 Opened by PID 1199 10.0.80.150 (the linux openvpn client): tap0 Link encap:Ethernet HWaddr 46:c2:27:c9:36:e3 inet addr:10.0.80.150 Bcast:10.0.80.255 Mask:255.255.255.0 inet6 addr: fe80::44c2:27ff:fec9:36e3/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:34336 errors:0 dropped:0 overruns:0 frame:0 TX packets:12951 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:11939564 (11.9 MB) TX bytes:1191746 (1.1 MB) 10.0.80.6 (the freebsd openvpn client): tap0: flags=8843 metric 0 mtu 1500 ether 00:bd:bf:f6:08:00 inet 10.0.80.6 netmask 0xffffff00 broadcast 10.0.80.255 Opened by PID 71953 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3328 bytes Desc: S/MIME Cryptographic Signature Url : http://lists.freebsd.org/pipermail/freebsd-cluster/attachments/20090408/03aca42c/smime.bin From sebster at sebster.com Wed Apr 8 07:49:42 2009 From: sebster at sebster.com (Sebastiaan van Erk) Date: Wed Apr 8 07:49:49 2009 Subject: CARP, openvpn in bridged mode, and ping In-Reply-To: <49DC7C96.2050203@sebster.com> References: <49DC7C96.2050203@sebster.com> Message-ID: <49DCB983.2070700@sebster.com> Hi, just to explain further, ARP does not seem to work in the following situation: client (tap0, 10.0.80.6) -> server (tap0 -> bridge0 -> em1, 10.0.80.77 -> carp1, 10.0.80.1) On the server I see on the em1 interface with tcpdump: 16:45:56.992297 00:bd:bf:f6:08:00 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 10.0.80.1 tell 10.0.80.6 but the server does not reply. If the client is directly on the em1 interface (another machine which is directly on the LAN and not via the bridge), the server DOES reply: 16:46:47.502250 00:0c:29:e2:46:c8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.80.1 tell 10.0.80.3 16:46:47.502283 00:0c:29:61:2a:55 > 00:0c:29:e2:46:c8, ethertype ARP (0x0806), length 42: arp reply 10.0.80.1 is-at 00:00:5e:00:01:02 Regards, Sebastiaan Sebastiaan van Erk wrote: > Hi, > > I have the following setup: two servers with a virtual LAN IP address > shared with CARP (the hosts are 10.0.80.77 and 10.0.80.76 and the > virtual IP address is 10.0.80.1). > > When I ping the VIP from any host on the LAN, it works fine. > > Next I have some openvpn clients (both 10.0.80.77 and 10.0.80.76 have > openvpn servers on their external IPs). The client IPs are on the LAN > using a bridge and are 10.0.80.150 (linux client) and 10.0.80.6 (freebsd > client). > > From linux I can ping the VIP (10.0.80.1) just fine, but when I do > arping I see (with tcpdump) that the the ARP requests are received by > the carp master on the tap0 device, but that it does not reply. > > From a FreeBSD VPN client I cannot ping the VIP (10.0.80.1), because it > does the ARP requests indefinitely and gets no answer. > > Both machines ping to the other hosts on the LAN just fine (e.g., all of > them can ping 10.0.80.77 just fine). > > Is there any way to get ARP to work (and thereby, ping to work) in this > configuration? > > Regards, > Sebastiaan > > PS: the relevant ifconfig info is: > > 10.0.80.77 (carp master and vpn server): > em1: flags=8943 metric 0 > mtu 1500 > options=98 > ether 00:0c:29:61:2a:55 > inet 10.0.80.77 netmask 0xffffff00 broadcast 10.0.80.255 > media: Ethernet autoselect (1000baseTX ) > status: active > bridge0: flags=8843 metric 0 mtu > 1500 > ether 12:d8:09:8d:1b:88 > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: tap0 flags=143 > ifmaxaddr 0 port 9 priority 128 path cost 2000000 > member: em1 flags=143 > ifmaxaddr 0 port 2 priority 128 path cost 20000 > carp1: flags=49 metric 0 mtu 1500 > inet 10.0.80.1 netmask 0xffffff00 > carp: MASTER vhid 174 advbase 1 advskew 0 > tap0: flags=8943 metric > 0 mtu 1500 > ether 00:bd:c0:02:00:00 > Opened by PID 1199 > > 10.0.80.150 (the linux openvpn client): > tap0 Link encap:Ethernet HWaddr 46:c2:27:c9:36:e3 > inet addr:10.0.80.150 Bcast:10.0.80.255 Mask:255.255.255.0 > inet6 addr: fe80::44c2:27ff:fec9:36e3/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:34336 errors:0 dropped:0 overruns:0 frame:0 > TX packets:12951 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:100 > RX bytes:11939564 (11.9 MB) TX bytes:1191746 (1.1 MB) > > 10.0.80.6 (the freebsd openvpn client): > tap0: flags=8843 metric 0 mtu 1500 > ether 00:bd:bf:f6:08:00 > inet 10.0.80.6 netmask 0xffffff00 broadcast 10.0.80.255 > Opened by PID 71953 > > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3328 bytes Desc: S/MIME Cryptographic Signature Url : http://lists.freebsd.org/pipermail/freebsd-cluster/attachments/20090408/00c47496/smime.bin From mailing at gaturkey.com Thu Apr 9 00:43:59 2009 From: mailing at gaturkey.com (Global Access Travel) Date: Thu Apr 9 00:44:35 2009 Subject: Private Shore Excursions-Turkey Message-ID: <5ac96c3f266b730658a62652e94e26cf@localhost.localdomain> [http://www.turkeycalling.us] PRIVATE SHORE EXCURSIONS- TURKEY Your cruise clients will make the best of their time in Turkey on a private shore excursion! Istanbul Kusadasi & Ephesus [mailto:incoming@gaturkey.com?subject=Private Shore Excursions- Turkey] **************************************************************************** Yasal Uyar?; Bu e-posta, sadece adreste belirtilen kisi veya kurulusun kullanimini hedeflemekte olup,mesajda yer alan bilgiler kisiye ozel ve gizli olabilir, yasalar ya da anlasmalar geregi ?c?nc? kisiler ile paylasilmasi m?mk?n olmayabilir.Mesaji alan kisi, mesajin g?nderilmek istendigi kisi veya kurulus degilse,bu mesaji yaymak,dagitmak veya kopyalamak yasaktir Mesaj tarafiniza yanlislikla ulasmissa l?tfen mesaji geri g?nderiniz ve sisteminizden siliniz. Global Turizm Hizmetleri Anonim Sirketi bu mesajin icerigi ile ilgili olarak hicbir hukuksal sorumlulugu kabul etmez. **************************************************************************** Disclaimer; This e-mail communication is intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential and that may not be made public by law or agreement. If the recipient of this message is not the intended recipient or entity, you are hereby notified that any further dissemination, distribution or copying of this information is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete it from your system. The Global Turizm Hizmetleri Anonim Sirketi does not accept legal responsibility for the contents of this message. *********************************************************************************************** Yasal Uyar?; Bu e-posta, sadece adreste belirtilen kisi veya kurulusun kullanimini hedeflemekte olup,mesajda yer alan bilgiler kisiye ozel ve gizli olabilir, yasalar ya da anlasmalar geregi ?c?nc? kisiler ile paylasilmasi m?mk?n olmayabilir.Mesaji alan kisi, mesajin g?nderilmek istendigi kisi veya kurulus degilse,bu mesaji yaymak,dagitmak veya kopyalamak yasaktir Mesaj tarafiniza yanlislikla ulasmissa l?tfen mesaji geri g?nderiniz ve sisteminizden siliniz. Global Turizm Hizmetleri Anonim Sirketi bu mesajin icerigi ile ilgili olarak hicbir hukuksal sorumlulugu kabul etmez. ********************************************************************************************** Disclaimer; This e-mail communication is intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential and that may not be made public by law or agreement. If the recipient of this message is not the intended recipient or entity, you are hereby notified that any further dissemination, distribution or copying of this information is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete it from your system. The Global Turizm Hizmetleri Anonim Sirketi does not accept legal responsibility for the contents of this message. This message was sent by: Global Access Incoming, Nuzhetiye cad, istanbul, besiktas 34357, Turkey Powered by iContact: http://freetrial.icontact.com To be removed click here: http://app.icontact.com/icp/mmail-mprofile.pl?r=46043618&l=82228&s=EL1A&m=562566&c=305227 Forward to a friend: http://app.icontact.com/icp/sub/forward?m=562566&s=46043618&c=EL1A&cid=305227 From sebster at sebster.com Mon Apr 20 12:21:15 2009 From: sebster at sebster.com (Sebastiaan van Erk) Date: Mon Apr 20 12:21:22 2009 Subject: pf and carp, BACKUP host dropping connection Message-ID: <49EC68B6.9090303@sebster.com> Hi, I have 3 hosts set up with 1 virtual IP using carp. I don't yet have pfsync (which I'm planning to do next). However, there is a strange behavior that I cannot understand. The 3 machines are all gateways between two networks and have 2 VIP ips which are used for routing (actually they have 4 networks and 4 VIPs, but only 2 are relevant in this case). When I ssh from one network to the other however, connections are sometimes blocked by pf. However, they're dropped on the machine which is NOT currently master! That is, I have machines: 1) carp1: flags=49 metric 0 mtu 1500 inet 10.0.80.74 netmask 0xffffff00 carp: MASTER vhid 2 advbase 1 advskew 0 carp3: flags=49 metric 0 mtu 1500 inet 10.0.82.74 netmask 0xffffff00 carp: MASTER vhid 4 advbase 1 advskew 0 2) carp0: flags=49 metric 0 mtu 1500 inet 212.61.136.74 netmask 0xfffffff0 carp: BACKUP vhid 1 advbase 1 advskew 50 carp2: flags=49 metric 0 mtu 1500 inet 10.0.81.74 netmask 0xffffff00 carp: BACKUP vhid 3 advbase 1 advskew 50 3) carp1: flags=49 metric 0 mtu 1500 inet 10.0.80.74 netmask 0xffffff00 carp: BACKUP vhid 2 advbase 1 advskew 100 carp3: flags=49 metric 0 mtu 1500 inet 10.0.82.74 netmask 0xffffff00 carp: BACKUP vhid 4 advbase 1 advskew 100 Then from the 10.0.80 network I do a ssh to the 10.0.82 network. The router for the 10.0.82 network is 10.0.82.74 and the router for the 10.0.80 network is 10.0.80.74 (the VIPs): > ssh 10.0.82.5 sebster@10.0.82.5's password: > Read from remote host 10.0.82.5: Connection reset by peer Connection to 10.0.82.5 closed. And then I get on the backup gateways pf log: machine 2: # tcpdump -nttteli pflog0 not src or dst port 6155 and not src or dst host 224.0.0.18 and not src or dst port 68 tcpdump: WARNING: pflog0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 96 bytes 000000 rule 11/0(match): block in on em1: 10.0.80.3.58876 > 10.0.82.5.22: [|tcp] 001161 rule 11/0(match): block in on em1: 10.0.80.3.58876 > 10.0.82.5.22: [|tcp] 000018 rule 11/0(match): block in on em1: 10.0.80.3.58876 > 10.0.82.5.22: tcp 20 [bad hdr length 0 - too short, < 20] machine 3: # tcpdump -nttteli pflog0 not src or dst port 6155 and not src or dst host 224.0.0.18 and not src or dst port 68 tcpdump: WARNING: pflog0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 96 bytes 000000 rule 11/0(match): block in on em1: 10.0.80.3.58876 > 10.0.82.5.22: [|tcp] 001113 rule 11/0(match): block in on em1: 10.0.80.3.58876 > 10.0.82.5.22: [|tcp] 000019 rule 11/0(match): block in on em1: 10.0.80.3.58876 > 10.0.82.5.22: tcp 20 [bad hdr length 0 - too short, < 20] I'm wondering why these backup hosts are blocking these packets, even though the master is still up, and why they are causing the connection to fail. (The pf on all 3 hosts do a "block return log on devif all" where devif is the interface with the real 10.0.80.x ip; however, why is it returning a RST packet when it's backup?). I think once I have pfsync the problem will go away due to the synchronized state (the backups won't block anymore), but it still seems strange to me that all 3 machines will then be actively filtering the packets... Does anybody know what's going on? Regards, Sebastiaan -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3328 bytes Desc: S/MIME Cryptographic Signature Url : http://lists.freebsd.org/pipermail/freebsd-cluster/attachments/20090420/0e7a52ee/smime.bin From stas at FreeBSD.org Tue Apr 21 08:37:36 2009 From: stas at FreeBSD.org (Stanislav Sedov) Date: Tue Apr 21 08:37:43 2009 Subject: pf and carp, BACKUP host dropping connection In-Reply-To: <49EC68B6.9090303@sebster.com> References: <49EC68B6.9090303@sebster.com> Message-ID: <20090421123735.f7caf3cc.stas@FreeBSD.org> On Mon, 20 Apr 2009 14:21:10 +0200 Sebastiaan van Erk mentioned: > Hi, > > I have 3 hosts set up with 1 virtual IP using carp. I don't yet have > pfsync (which I'm planning to do next). However, there is a strange > behavior that I cannot understand. > > The 3 machines are all gateways between two networks and have 2 VIP ips > which are used for routing (actually they have 4 networks and 4 VIPs, > but only 2 are relevant in this case). When I ssh from one network to > the other however, connections are sometimes blocked by pf. However, > they're dropped on the machine which is NOT currently master! > > That is, I have machines: > > 1) > carp1: flags=49 metric 0 mtu 1500 > inet 10.0.80.74 netmask 0xffffff00 > carp: MASTER vhid 2 advbase 1 advskew 0 > carp3: flags=49 metric 0 mtu 1500 > inet 10.0.82.74 netmask 0xffffff00 > carp: MASTER vhid 4 advbase 1 advskew 0 > > 2) > carp0: flags=49 metric 0 mtu 1500 > inet 212.61.136.74 netmask 0xfffffff0 > carp: BACKUP vhid 1 advbase 1 advskew 50 > carp2: flags=49 metric 0 mtu 1500 > inet 10.0.81.74 netmask 0xffffff00 > carp: BACKUP vhid 3 advbase 1 advskew 50 > > 3) > carp1: flags=49 metric 0 mtu 1500 > inet 10.0.80.74 netmask 0xffffff00 > carp: BACKUP vhid 2 advbase 1 advskew 100 > carp3: flags=49 metric 0 mtu 1500 > inet 10.0.82.74 netmask 0xffffff00 > carp: BACKUP vhid 4 advbase 1 advskew 100 > > > Then from the 10.0.80 network I do a ssh to the 10.0.82 network. The > router for the 10.0.82 network is 10.0.82.74 and the router for the > 10.0.80 network is 10.0.80.74 (the VIPs): > > > ssh 10.0.82.5 > sebster@10.0.82.5's password: > > Read from remote host 10.0.82.5: Connection reset by peer > Connection to 10.0.82.5 closed. > > And then I get on the backup gateways pf log: > > machine 2: > # tcpdump -nttteli pflog0 not src or dst port 6155 and not src or dst > host 224.0.0.18 and not src or dst port 68 > tcpdump: WARNING: pflog0: no IPv4 address assigned > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size > 96 bytes > 000000 rule 11/0(match): block in on em1: 10.0.80.3.58876 > > 10.0.82.5.22: [|tcp] > 001161 rule 11/0(match): block in on em1: 10.0.80.3.58876 > > 10.0.82.5.22: [|tcp] > 000018 rule 11/0(match): block in on em1: 10.0.80.3.58876 > > 10.0.82.5.22: tcp 20 [bad hdr length 0 - too short, < 20] > > machine 3: > # tcpdump -nttteli pflog0 not src or dst port 6155 and not src or dst > host 224.0.0.18 and not src or dst port 68 > tcpdump: WARNING: pflog0: no IPv4 address assigned > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size > 96 bytes > 000000 rule 11/0(match): block in on em1: 10.0.80.3.58876 > > 10.0.82.5.22: [|tcp] > 001113 rule 11/0(match): block in on em1: 10.0.80.3.58876 > > 10.0.82.5.22: [|tcp] > 000019 rule 11/0(match): block in on em1: 10.0.80.3.58876 > > 10.0.82.5.22: tcp 20 [bad hdr length 0 - too short, < 20] > > I'm wondering why these backup hosts are blocking these packets, even > though the master is still up, and why they are causing the connection > to fail. (The pf on all 3 hosts do a "block return log on devif all" > where devif is the interface with the real 10.0.80.x ip; however, why is > it returning a RST packet when it's backup?). > > I think once I have pfsync the problem will go away due to the > synchronized state (the backups won't block anymore), but it still seems > strange to me that all 3 machines will then be actively filtering the > packets... > > Does anybody know what's going on? > I'd suggest to look first why all of them're receiving this traffic. It looks like something is not right in the network itself. -- Stanislav Sedov ST4096-RIPE !DSPAM:49ed85cd967001477745436! From sebster at sebster.com Tue Apr 21 21:01:06 2009 From: sebster at sebster.com (Sebastiaan van Erk) Date: Tue Apr 21 21:01:12 2009 Subject: pf and carp, BACKUP host dropping connection In-Reply-To: <20090421123735.f7caf3cc.stas@FreeBSD.org> References: <49EC68B6.9090303@sebster.com> <20090421123735.f7caf3cc.stas@FreeBSD.org> Message-ID: <49EE340E.4090006@sebster.com> Stanislav Sedov wrote: > On Mon, 20 Apr 2009 14:21:10 +0200 > Sebastiaan van Erk mentioned: >> I think once I have pfsync the problem will go away due to the >> synchronized state (the backups won't block anymore), but it still seems >> strange to me that all 3 machines will then be actively filtering the >> packets... >> >> Does anybody know what's going on? >> > > I'd suggest to look first why all of them're receiving this traffic. It > looks like something is not right in the network itself. After reading about CARP some more, I think that's the expected behavior: --- http://www.openbsd.org/faq/faq6.html#CARP --- How it works: CARP is a multicast protocol. It groups several physical computers together under one or more virtual addresses. Of these, one system is the master and responds to all packets destined for the group, the other systems act as hot spares. --- http://www.openbsd.org/faq/faq6.html#CARP --- Since I don't have pfsync enabled yet the other two machines don't have the propper state and will drop the connection. Normally this would only pollute the log, but on the internal networks I don't want connections to hang for long periods so I do "block return". This causes pf to respond to the traffic since it doesn't know anything about the machine being a carp backup, and thus the originating host receives a RST and drops the connection. I'm wondering if the combination block return + carp is going to work at all, even with pfsync... I will do some more research on that. Regards, Sebastiaan -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3328 bytes Desc: S/MIME Cryptographic Signature Url : http://lists.freebsd.org/pipermail/freebsd-cluster/attachments/20090421/0cc96273/smime.bin