Weird networking issue

[Glen Barber]

On Mon, Aug 3, 2009 at 5:16 PM, Joel Dahl<joel at vnode.se> wrote:
> Yesterday I noticed something weird with one of my mail servers (FreeBSD
> 7.2), which is located far away from me.  At first I thought it was my mail
> client that was misbehaving since a couple of mails that I retrieved from
> the server looked...odd.  Characters were missing, and the mail headers were
> sometimes (perhaps 1 out of 100 mails) sort of scrambled.
>
> This server isn't doing anything really useful, except handling a few mails
> every day.  No load at all and it has been running for a few months without
> any problems.
>
> Anyway, after some time I realized that Thunderbird probably wasn't the
> problem.  I discovered that ssh connections suddenly dropped (seems random)
> when I was connected to the server.  Often with messages like:
>
> Received disconnect from x.x.x.x: 2: Bad packet length 3365265859.
>
> I tried to ping it and I constantly got 5-15% packet loss.  The same
> symptoms are there if I connect to it from other places, so my home
> connection isn't the problem.  I get 0 packet loss if I ping the gateway
> which the server is behind from my home connection.
>
> Anyway, while I was connected to the machine (and scratching my head) I did
> a csup from the server to fetch ports and other stuff, but that failed every
> time, with different messages:
>
> 1st try:
>
>  Edit ports/Mk/bsd.port.mk
>  Add delta 1.625 2009.08.03.15.36.58 miwi
>  Edit ports/UPDATING
>  Add delta 1.845 2009.08.03.19.08.48 dougb
> csup: inflate: invalid stored block lengths
>
> 2nd try:
>
>  Edit ports/devel/libvc/Makefile
>  Add delta 1.11 2009.08.02.19.33.28 mezz
>  Edit ports/devel/libytnef/Makefile
>  Add delta 1.6 2009.08.02.19.33.28 mezz
> Receiver: Protocol error
>
> 3rd try:
>
> Edit ports/devel/p5-threads-shared/distinfo
>  Add delta 1.4 2009.07.22.17.25.14 pgollucci
>  Edit ports/devel/p5-usb/pkg-plist
>  Add delta 1.3 2009.07.30.23.04.41 pgollucci
> csup: inflate: invalid distance too far back
>
> Ok.  So something is messed up.
>
> Checked my other server (located at the same place) to see if there were any
> problems (it is running VMware ESXI with 4 virtual machines, all running
> FreeBSD 7.2.), but all of the virtual hosts were fine.  No packet loss, no
> ssh problems, no nothing.  This server is connected to the same switch and
> behind the same gateway.  So are also a bunch of Windows servers.  None of
> them have any problems.
>
> Checked the logs on my mail server to see if there were anything weird...and
> the logs were filled with this:
>
> arp: x.x.x.x moved from 00:16:c7:24:69:ae to 00:16:cf:24:69:ae on bge0
> arp: x.x.x.x moved from 00:16:cf:24:69:ae to 00:16:c7:24:69:ae on bge0
> arp: x.x.x.x moved from 00:16:c7:24:69:ae to 00:16:cf:24:69:ae on bge0
> arp: x.x.x.x moved from 00:16:cf:24:69:ae to 00:16:c7:24:69:ae on bge0
> arp: x.x.x.x moved from 00:16:c7:24:69:ae to 00:16:cf:24:69:ae on bge0
> arp: x.x.x.x moved from 00:16:cf:24:69:ae to 00:16:c7:24:69:ae on bge0
> arp: x.x.x.x moved from 00:16:c7:24:69:ae to 00:16:cf:24:69:ae on bge0
> arp: x.x.x.x moved from 00:16:cf:24:69:ae to 00:16:c7:24:69:ae on bge0
>
> x.x.x.x is my gateway which all my servers are behind.  These messages
> started yesterday, about the same time that the problems appeared.
>
> Any ideas? :-)
>

I have 2:
1.) Bad NIC
2.) Bad CPU

A while back, I found out I had a bad CPU after replacing everything
else on the machine -- I would still receive CRC mismatch errors with
portsnap(8).

The NIC is more probable IMHO, but both should be suspect here.

-- 
Glen Barber