ARP behavior in FreeBSD vs Linux

[Gary Palmer]

Pieter de Boer wrote:

>>> Is there any advantage/disadvantage in ARP implementation on FreeBSD
>>> over that of Linux? Thanks.
>>
> I was unhappily surprised by this 'feature'. I find it pretty 
> counter-intuitive. I expect two interfaces to be seperated inside a 
> kernel, but Linux more or less binds them together. Incoming traffic 
> on the 'wrong' interface will gladly be accepted, too. This broke 
> things for me, because I didn't want to have that certain IP-address 
> accessible.
>
> That said, this happens only when you have two interfaces connected to 
> the same subnet, which is a bit evil anyhow. It may be beneficial for 
> Linux to do things this way, perhaps for redundancy-purposes (two 
> interfaces, one IP-address, IP reachable over both interfaces, when 
> one fails, the other takes over.. no idea if that works out-of-the-box).


There is another side effect, which comes into view with certain 
configurations behind load balancers.  Foundry has an option (I believe 
called "DSR" for Direct Server Return) which just fiddles with the MAC 
address of the destination.  Other companies load balancers will 
probably have the same option, but I've no idea what they'll call it. 
For the connection to be accepted, all servers which are expected to 
answer for a particular load balanced IP address have to have that IP 
configured on one of their interfaces, typically loopback.  The host 
sees that the connection is for one of its interfaces, accepts the 
connection and life is happy.  The return path from the host to the 
originator bypasses the load balancer, and effectively halves the 
traffic that the LB is having to process and do table lookups on, etc.  
This obviously greatly increases the available capacity of the LB.

With a Linux box answering ARP as described above, it is possible that 
the upstream router (or routers) COULD learn that the load balanced IP 
actually belongs on one of the servers rather than the load balancer.  
If that happens, your load balanced farm will quickly degrade and you'll 
be scratching your head for hours to try and figure out whats going on.  
Or the LB and the Linux box will get into an ARP war and random TCP 
connections will get RSTs from the Linux box.