Network problem

[Mario Lobo]

Hi;

I have 2 links with 2 different ISPs on this firewall.

On one link, everything is normal

[~]>ping www.google.com
PING www.google.com (64.233.185.104): 56 data bytes
64 bytes from 64.233.185.104: icmp_seq=0 ttl=44 time=175.683 ms
64 bytes from 64.233.185.104: icmp_seq=1 ttl=44 time=174.499 ms
64 bytes from 64.233.185.104: icmp_seq=2 ttl=44 time=171.770 ms
64 bytes from 64.233.185.104: icmp_seq=3 ttl=44 time=170.737 ms
64 bytes from 64.233.185.104: icmp_seq=4 ttl=44 time=170.705 ms
64 bytes from 64.233.185.104: icmp_seq=5 ttl=44 time=171.699 ms
64 bytes from 64.233.185.104: icmp_seq=6 ttl=44 time=178.248 ms
64 bytes from 64.233.185.104: icmp_seq=7 ttl=44 time=170.924 ms
64 bytes from 64.233.185.104: icmp_seq=8 ttl=44 time=174.189 ms
64 bytes from 64.233.185.104: icmp_seq=9 ttl=44 time=169.742 ms
64 bytes from 64.233.185.104: icmp_seq=10 ttl=44 time=171.725 ms

on the other, I get this:

[~]>setfib 1 ping www.google.com
PING www.google.com (64.233.185.104): 56 data bytes
ping: sendto: No buffer space available
64 bytes from 64.233.185.104: icmp_seq=0 ttl=39 time=2837.234 ms
64 bytes from 64.233.185.104: icmp_seq=2 ttl=39 time=1999.021 ms
ping: sendto: No buffer space available
64 bytes from 64.233.185.104: icmp_seq=3 ttl=39 time=2256.407 ms
64 bytes from 64.233.185.104: icmp_seq=4 ttl=39 time=3002.886 ms
64 bytes from 64.233.185.104: icmp_seq=6 ttl=39 time=1570.676 ms
64 bytes from 64.233.185.104: icmp_seq=7 ttl=39 time=1765.401 ms
64 bytes from 64.233.185.104: icmp_seq=8 ttl=39 time=2148.457 ms
ping: sendto: No buffer space available
64 bytes from 64.233.185.104: icmp_seq=9 ttl=39 time=2207.230 ms
64 bytes from 64.233.185.104: icmp_seq=10 ttl=39 time=1752.444 ms
64 bytes from 64.233.185.104: icmp_seq=12 ttl=39 time=1619.150 ms

re0 - PROD-LAN
re1 - LAB-LAN
sk0 - ISP1 (fib 0)
sk1 - ISP2 (fib 1)

FreeBSD FreeBSD 8.3-STABLE #0 r204106 Wed Sep 17 10:28:45 BRT 2014

last pid:  6963;  load averages:  0.00,  0.00,0.00   
19:47:29 61 processes:  1 running, 60 sleeping 
CPU:  0.0% user,  0.0% nice,  0.2% system,  0.1% interrupt,  99.7% idle 
Mem: 210M Active, 
     3933M Inact, 
     1559M Wired, 
     516K Cache, 
     771M Buf, 
     1671M Free 
Swap: 16G Total, 16G Free

This was at 7:oo PM but even during peak work ours it doesn't go lower
than 88% idle when it peaks usage.

I have one squid for each link with exactly the same configs and pf
doing the round-robin between the two.

This server has never done this before!. I've been googling around and
tried a few suggestions but none really solved it.

I have already set kern.ipc.nmbclusters from 32768 to 131072. It seemed
to delay the "no buffer" message but it's back.

My stats seem to me to be well bellow the limits.

[~]>netstat -m
2187/2178/4365 mbufs in use (current/cache/total)
1861/1453/3314/131072 mbuf clusters in use (current/cache/total/max)
1861/1339 mbuf+clusters out of packet secondary zone in use
(current/cache) 212/321/533/65536 4k (page size) jumbo clusters in use
(current/cache/total/max) 0/0/0/32768 9k jumbo clusters in use
(current/cache/total/max) 0/0/0/16384 16k jumbo clusters in use
(current/cache/total/max) 5169K/4734K/9903K bytes allocated to network
(current/cache/total) 0/0/0 requests for mbufs denied
(mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied
(4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
3 requests for I/O initiated by sendfile
0 calls to protocol drain routines

I already rebooted and switched cables. The only thing I didnt do yet
was to switch boards. Should I be looking into that?

Any advice?

Thanks,
-- 
Mario Lobo
http://www.mallavoodoo.com.br
FreeBSD since 2.2.8 [not Pro-Audio.... YET!!] (99% winblows FREE)
 
"UNIX was not designed to stop you from doing stupid things, 
because that would also stop you from doing clever things."