Looking For Ideas or Suggestions

[Greg Rowe]

Greetings, 
 I've been chasing a network interface "freeze" problem on and off for some 
time now and it's driving me nuts ! 

The problem occurs on two identical mail servers that sit behind a firewall. 
Both systems have two ethernet interfaces and when I'm having this problem 
the external interface will "freeze" once or twice an hour for between 10-15 
seconds. The systems continue to run during these freezes and it doesn't 
effect the traffic on the 2nd interface. The problem is also intermittent in 
that it will effect one system for several weeks and then just go away. Today 
it's effecting both systems.

 The systems are Sun Fire V60X dual  3.06GHZ Xeon processor systems with 
integrated Intel PRO/1000 (em0) ethernet ports, 2GB of memory. We have a 
number of these systems and these are the only two experiencing the problem. 
They are running 4.11 STABLE although they were originally installed with 
4.10 STABLE and upgraded to see if this fixed the problem. The one system 
currently has an Intel EtherExpress Pro/100B card installed as the primary 
interface to see if the em0 was my problem, but I still have freezes using 
fxp0. Both systems are very lightly loaded and running Sendmail and anti-spam 
packages.

 The systems hang off Catalyst switches that have been checked and rechecked. 
No errors or config issues. Duplex, speed and mediaopt are all set in rc.conf 
and aren't autodetected. Cables and ports have all been swapped. No errors in 
netstat or any logs. Sysctl " log_in_vain's" aren't showing me any clues. The 
interface just freezes and then starts again with no messages. Tests using 
pings from system to the other out each interface prove that the emo/fxp0 
freezes with packet loss while pings to the em1 interface have no problems.

 Now, here's where it gets stranger. By accident I found one way to guarantee 
that a freeze won't occur. If I log into the system via the fxp0/em0 
interface and start a ping against the IP of that interface. As long as the 
ping is running (I've tried days) and outputing the ping stats every second, 
the interface is freeze free ! I liken it to keeping the interface "warm" !! 
Doing the same ping with a -q for some reason doesn't stop the freezes. It 
needs the ping output to keep "warm". Pinging the em0 address from another 
system or while logged in through the other interface also won't stop the 
freeze. The freeze isn't login window related, although it may sound that 
way. The interface just stops working for no apparent reason and then starts 
again after 10 or 15 seconds.

 I've gone the network sniffer route and really can't see anything out of the 
ordinary happening when the freeze occurs. Most ports are blocked by the 
firewall and the systems also have ipfw enabled (taken out of the kernel on 
one to see if maybe that was causing the problem). I'm running out of ideas 
short of replacing bigger hardware than an ethernet card. The problem is I 
don't know what to replace. I've been building and running FreeBSD systems 
for many years and this one has me and everyone else stumped.
 
 I'm looking for any suggestions as to what I could enable or tweak that may 
give me some info as to why the interfaces are intermittently freezing. I'm 
willing to try just about anything right now. Thanks.