ARP not working on secondary interface

[Rohit Atri (ratri)]

This query here talks about a similar scenario - https://forums.freebsd.org/threads/26050/
Only difference being that in my case, ARP replies are actually received.

Thanks,
Rohit



On 07/12/16, 1:12 PM, "owner-freebsd-net at freebsd.org on behalf of Rohit Atri (ratri)" <owner-freebsd-net at freebsd.org on behalf of ratri at cisco.com> wrote:

>Hello,
>
>I have EC2 instance with a custom FreeBSD kernel (9.2-RELEASE) with two interfaces – nic0 & nic1. Both have elastic IPs associated  - E0 & E1.
>
>I noticed that when I ping E1, the ICMP response packets go out of nic1 instead of nic0 & thus, from the ping source perspective, host is down. To workaround the asymmetric routing, I added a default route via nic1 to FIB 1 & added an IPFW rule to ensure FIB 1 is used for all packets received via nic1.
>
>Now, when nic1 tries to respond to an ICMP request, the first thing it does is to ARP for the router’s IP. The router responds with its MAC address but nic1 doesn’t seem to be processing it. The entry for default router’s IP in the ARP table remains marked ‘incomplete’ and never gets updated -
>
>*********************
>
># tcpdump -ni nic1 -p arp
>
>07:29:04.922209 ARP, Request who-has 172.31.16.1 tell 172.31.30.133, length 28
>
>07:29:04.922316 ARP, Reply 172.31.16.1 is-at 02:18:c9:53:d0:2f, length 42
>
>
># arp -a
>
>ip-172-31-16-1.us-west-2.compute.internal (172.31.16.1) at (incomplete) on nic1 expired [ethernet]
>
>ip-172-31-30-133.us-west-2.compute.internal (172.31.30.133) at 02:6e:32:90:0e:ad on nic1 permanent [ethernet]
>
>ip-172-31-16-1.us-west-2.compute.internal (172.31.16.1) at 02:18:c9:53:d0:2f on nic0 expires in 1199 seconds [ethernet]
>
>ip-172-31-22-113.us-west-2.compute.internal (172.31.22.113) at 02:ea:39:ce:5a:49 on nic0 permanent [ethernet]
>
>*********************
>But… if I delete the arp entry (arp –d 172.31.16.1), ARP entry for the default router IP gets updated properly -
>
>
>*********************
>
># arp -a
>
>ip-172-31-16-1.us-west-2.compute.internal (172.31.16.1) at 02:18:c9:53:d0:2f on nic1 expires in 1195 seconds [ethernet]
>
>…
>
>*********************
>
>But but… this only lasts until the entry expires. After that, the entry gets marked ‘incomplete’ again.
>
>
>I also notice these ARP requests start being received from the router around the same time I delete the entry (may the source address of these packets are being used to update the ARP entry?) -
>
>*********************
>
>07:32:41.411229 ARP, Request who-has 172.31.16.1 tell 172.31.30.133, length 28
>
>07:32:41.411297 ARP, Reply 172.31.16.1 is-at 02:18:c9:53:d0:2f, length 42
>
>07:33:04.482719 ARP, Request who-has 172.31.30.133 tell 172.31.16.1, length 42 <<<<<<
>
>07:33:04.482734 ARP, Reply 172.31.30.133 is-at 02:6e:32:90:0e:ad, length 28
>
>*********************
>
>
>Any idea why the ARP replies are not being processed by nic1? Any known workarounds?
>
>
>Thanks,
>
>Rohit
>
>
>
>PS –
>
>FIB 1:
>
>Internet:
>
>Destination        Gateway            Flags    Refs      Use  Netif Expire
>
>default            172.31.16.1        UGS         0     1985   nic1
>
>127.0.0.1          link#2             UH          0        0    lo0
>
>172.31.16.0/20     link#4             U           0        0   nic1
>
>
>IFPW -
>
>00002 setfib 1 ip from any to any via nic1
>
>65535 allow ip from any to any
>_______________________________________________
>freebsd-net at freebsd.org mailing list
>https://lists.freebsd.org/mailman/listinfo/freebsd-net
>To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"