"arp: moved" messages with bogus MAC addresses

[Christian Ullrich]

* Matthew Seaman <matthew at FreeBSD.org> wrote:

> On 10/31/16 08:43, Christian Ullrich wrote:

> > I see a lot of messages like these in the logs of my 11-STABLE VMs
> > (running on VMware ESXi 6):
> > 
> > arp: 192.168.92.20 moved from 20:90:f1:53:01:f8 to 00:90:fb:1b:7d:cd on em0
> > arp: 192.168.92.20 moved from 20:90:f1:53:01:f8 to 00:90:fb:1b:7d:cd on em0
> > arp: 192.168.92.20 moved from 20:90:f1:53:01:f8 to 00:90:fb:1b:7d:cd on em0
> > arp: 192.168.92.20 moved from 20:90:f1:53:01:f8 to 00:90:fb:1b:7d:cd on em0
> > 
> > I have no idea what causes these. The "to" MAC is correct for this IP,
> > but the "from" is completely bogus. I have tcpdump'ed the network until
> > two of these logs appeared, and the 20:90:f1 address never showed up; in
> > fact, the byte pattern did not appear once anywhere in the dump. This
> > should exclude the possibility of some weird VMware-related thing going
> > on.
> 
> I see exactly the same thing with VirtualBox -- this is something to do
> with the way these virtualization systems provide network access within
> the host system.
> 
> Normally this message would indicate that two machines on your network
> had been setup with the same IP number,  so it's generally a good thing
> to have logged.  In this case however, the IP switching around between
> MAC addresses seems to be normal for the virtualization environment.

Yes, I would agree with that, *if* the bogus MAC addresses were
actually visible on the interface. In my case, they are not, and even
more confusingly, the MACs never move the other way. How could the
implementation of the virtual NIC, or the processing of network
packets in the virtualization engine (trying not to say "hypervisor"
here) affect the content of the kernel's ARP cache?

OTOH, as far as I can tell it never tries to send anything *to* the
questionable address either, as though that had never been in the ARP
cache to begin with.

I'm going to put a kernel that shows this effect on some physical
hardware; if it does not happen there, at least I have a clearer
indication of where the problem is.

This is actually not the only unexplained low-level network issue I
have right now; one of the three systems also does this (consecutive
lines):

Oct 28 11:51:08 infra2 kernel: Limiting closed port RST response from
222 to 200 packets/sec
Oct 28 11:51:09 infra2 kernel: Limiting closed port RST response from
275 to 200 packets/sec
Oct 28 12:12:39 infra2 kernel: Limiting closed port RST response from
367 to 200 packets/sec
Oct 28 12:34:23 infra2 kernel: Limiting closed port RST response from
315 to 200 packets/sec

# sysctl net.inet.tcp.log_in_vain
net.inet.tcp.log_in_vain: 1

So if there *was* such a packet storm on a closed port, it should show
up as a ton of "tcp_input: Connection attempt to closed port" messages
somewhere in between. Needless to say that (while there are a few of
them) I also cannot see anything like this number of SYNs (or RSTs for
that matter) on the interface, so whatever is going on here is 
apparently inventing both ARP and TCP packets out of thin air.

-- 
Christian