ARP not working on secondary interface

[Rohit Atri (ratri)]

Hello,

I have EC2 instance with a custom FreeBSD kernel (9.2-RELEASE) with two interfaces – nic0 & nic1. Both have elastic IPs associated  - E0 & E1.

I noticed that when I ping E1, the ICMP response packets go out of nic1 instead of nic0 & thus, from the ping source perspective, host is down. To workaround the asymmetric routing, I added a default route via nic1 to FIB 1 & added an IPFW rule to ensure FIB 1 is used for all packets received via nic1.

Now, when nic1 tries to respond to an ICMP request, the first thing it does is to ARP for the router’s IP. The router responds with its MAC address but nic1 doesn’t seem to be processing it. The entry for default router’s IP in the ARP table remains marked ‘incomplete’ and never gets updated -

*********************

# tcpdump -ni nic1 -p arp

07:29:04.922209 ARP, Request who-has 172.31.16.1 tell 172.31.30.133, length 28

07:29:04.922316 ARP, Reply 172.31.16.1 is-at 02:18:c9:53:d0:2f, length 42


# arp -a

ip-172-31-16-1.us-west-2.compute.internal (172.31.16.1) at (incomplete) on nic1 expired [ethernet]

ip-172-31-30-133.us-west-2.compute.internal (172.31.30.133) at 02:6e:32:90:0e:ad on nic1 permanent [ethernet]

ip-172-31-16-1.us-west-2.compute.internal (172.31.16.1) at 02:18:c9:53:d0:2f on nic0 expires in 1199 seconds [ethernet]

ip-172-31-22-113.us-west-2.compute.internal (172.31.22.113) at 02:ea:39:ce:5a:49 on nic0 permanent [ethernet]

*********************
But… if I delete the arp entry (arp –d 172.31.16.1), ARP entry for the default router IP gets updated properly -


*********************

# arp -a

ip-172-31-16-1.us-west-2.compute.internal (172.31.16.1) at 02:18:c9:53:d0:2f on nic1 expires in 1195 seconds [ethernet]

…

*********************

But but… this only lasts until the entry expires. After that, the entry gets marked ‘incomplete’ again.


I also notice these ARP requests start being received from the router around the same time I delete the entry (may the source address of these packets are being used to update the ARP entry?) -

*********************

07:32:41.411229 ARP, Request who-has 172.31.16.1 tell 172.31.30.133, length 28

07:32:41.411297 ARP, Reply 172.31.16.1 is-at 02:18:c9:53:d0:2f, length 42

07:33:04.482719 ARP, Request who-has 172.31.30.133 tell 172.31.16.1, length 42 <<<<<<

07:33:04.482734 ARP, Reply 172.31.30.133 is-at 02:6e:32:90:0e:ad, length 28

*********************


Any idea why the ARP replies are not being processed by nic1? Any known workarounds?


Thanks,

Rohit



PS –

FIB 1:

Internet:

Destination        Gateway            Flags    Refs      Use  Netif Expire

default            172.31.16.1        UGS         0     1985   nic1

127.0.0.1          link#2             UH          0        0    lo0

172.31.16.0/20     link#4             U           0        0   nic1


IFPW -

00002 setfib 1 ip from any to any via nic1

65535 allow ip from any to any