bhyve and arp problem

[Daniel Braniss]

> On 2 Apr 2018, at 16:51, Rodney W. Grimes <freebsd-rwg at pdx.rh.CN85.dnsmgr.net> wrote:
> 
>>> On 2 Apr 2018, at 15:33, Harry Schmalzbauer <freebsd at omnilan.de> wrote:
>>> 
>>> Bez?glich Daniel Braniss's Nachricht vom 30.03.2018 13:16 (localtime):
>>>> hi,
>>>> this is my first attempt at bhyve, and so far all seems ok, except
>>>> in my guest, the mac address of the hosting keeps flipping, ie, every 20 minutes
>>>> i see a message :
>>>> 	? arp: nnn (the hosting ip)  moved from xxxx to yyyy
>>>> on both the host and guest I?m running a very resent -stable.
>>>> the yyyy is the mac of the host nic, while the xxxx is the tap0
>>>> 
>>>> i know this looks harmless, but it?s annoying
>>> 
>>> You can calm it with
>>> 'sysctl net.link.ether.inet.log_arp_movements=0'
>>> 
>>> There's also "net.link.ether.inet.log_arp_wrong_iface" and
>>> "net.inet.ip.check_interface" which influence related behaviour.
>>> 
>>> You also posted (documentationized IP-addresses):
>>>> I think the problem starts with the host seeing the client/guest on 2 interfaces, the nic (mlnxen0) and the tap(tap0)
>>>> on the host:
>>>> 
>>>> arp -a
>>>> ...
>>>> bhv-00.cs.huji.ac.il (192.0.2.246) at xx.xx.xx.xx.xx on tap0 expires in 1001 seconds [ethernet]
>>>> bhv-00.cs.huji.ac.il (192.0.2.246) at xx.xx.xx.xx.xx on mlxen0 expires in 644 seconds [ethernet]
>>> 
>> 
>> the above 2 lines are on the host running bhyve (server?) and the MACs belong to the client, and they are identical,
>> there is no complaints.
>> (BTW, did you change the ip?s?)
>> 
>>> Initially, you reference two MAC-addresses with xxxx and yyyy.
>> this is on the client, where the MAC are different (it?s of the hosting computer).
>> 
>>> The recent post indicates non-different MAC-addresses.
>>> 
>>> If xxxx and yyyy - resp. xx.xx.xx.xx.xx - are equal (but seen on
>>> different interfaces), this wouldn't get logged I think.
>>> But it was the only harmless case for straight forward setups.
>>> Even with STP/LACP/CARP/etc. in place, "arp: IP-address moved" always
>>> indicates a misconfiguration and I don't know any example where the two
>>> different MAC-Addresses for one IP-address were harmless.
>>> While using a single (locally administrated?) MAC address more than once
>>> sitewide _can_ make sense, having two interfaces on one host which both
>>> are on the same ethernet segment like the two interfaces with the same
>>> MAC address, looks like an unintended setup.
>>> 
>>> So I strongly suggest to analyze your setup before altering the
>>> mentioned sysctl!!!
>>> 
>> I do want to know if there are ip/mac issues, it usually happens when more than one host has the same ip,
>> which is not the case here :-(
> 
> Are you trying to use the HOSTS ip address in the GUEST?

the client is using the server’s /usr/local, which is mounted via nfs.
so I guess the answer is yes.

> And how do you have an mlxen interface in a GUEST?
no
the guest has only vtnet0 and lo0
the ip of the client is obtained via dhcp
on the server, there is a bridge, bridge0 and it bridges between the taps and the mxlen0

> Is this being done with PCI passthrough?
again, no.

cheers,
	danny

> 
> 
> -- 
> Rod Grimes                                                 rgrimes at freebsd.org