Weird networking issue

[Joel Dahl]

Yesterday I noticed something weird with one of my mail servers (FreeBSD 7.2), 
which is located far away from me.  At first I thought it was my mail client 
that was misbehaving since a couple of mails that I retrieved from the server 
looked...odd.  Characters were missing, and the mail headers were sometimes 
(perhaps 1 out of 100 mails) sort of scrambled.

This server isn't doing anything really useful, except handling a few mails 
every day.  No load at all and it has been running for a few months without 
any problems.

Anyway, after some time I realized that Thunderbird probably wasn't the 
problem.  I discovered that ssh connections suddenly dropped (seems random) 
when I was connected to the server.  Often with messages like:

Received disconnect from x.x.x.x: 2: Bad packet length 3365265859.

I tried to ping it and I constantly got 5-15% packet loss.  The same symptoms 
are there if I connect to it from other places, so my home connection isn't 
the problem.  I get 0 packet loss if I ping the gateway which the server is 
behind from my home connection.

Anyway, while I was connected to the machine (and scratching my head) I did a 
csup from the server to fetch ports and other stuff, but that failed every 
time, with different messages:

1st try:

  Edit ports/Mk/bsd.port.mk
   Add delta 1.625 2009.08.03.15.36.58 miwi
  Edit ports/UPDATING
   Add delta 1.845 2009.08.03.19.08.48 dougb
csup: inflate: invalid stored block lengths

2nd try:

  Edit ports/devel/libvc/Makefile
   Add delta 1.11 2009.08.02.19.33.28 mezz
  Edit ports/devel/libytnef/Makefile
   Add delta 1.6 2009.08.02.19.33.28 mezz
Receiver: Protocol error

3rd try:

Edit ports/devel/p5-threads-shared/distinfo
   Add delta 1.4 2009.07.22.17.25.14 pgollucci
  Edit ports/devel/p5-usb/pkg-plist
   Add delta 1.3 2009.07.30.23.04.41 pgollucci
csup: inflate: invalid distance too far back

Ok.  So something is messed up.

Checked my other server (located at the same place) to see if there were any 
problems (it is running VMware ESXI with 4 virtual machines, all running 
FreeBSD 7.2.), but all of the virtual hosts were fine.  No packet loss, no ssh 
problems, no nothing.  This server is connected to the same switch and behind 
the same gateway.  So are also a bunch of Windows servers.  None of them have 
any problems.

Checked the logs on my mail server to see if there were anything weird...and 
the logs were filled with this:

arp: x.x.x.x moved from 00:16:c7:24:69:ae to 00:16:cf:24:69:ae on bge0
arp: x.x.x.x moved from 00:16:cf:24:69:ae to 00:16:c7:24:69:ae on bge0
arp: x.x.x.x moved from 00:16:c7:24:69:ae to 00:16:cf:24:69:ae on bge0
arp: x.x.x.x moved from 00:16:cf:24:69:ae to 00:16:c7:24:69:ae on bge0
arp: x.x.x.x moved from 00:16:c7:24:69:ae to 00:16:cf:24:69:ae on bge0
arp: x.x.x.x moved from 00:16:cf:24:69:ae to 00:16:c7:24:69:ae on bge0
arp: x.x.x.x moved from 00:16:c7:24:69:ae to 00:16:cf:24:69:ae on bge0
arp: x.x.x.x moved from 00:16:cf:24:69:ae to 00:16:c7:24:69:ae on bge0

x.x.x.x is my gateway which all my servers are behind.  These messages started 
yesterday, about the same time that the problems appeared.

Any ideas? :-)

--
Joel