network problems 7.0-p3: sendto: Operation not permitted

[Alex Trull]

Robert,

The config files you attached were a series of 403 forbidden htmls.

The icmp pings (1 per second) do not constitute an attack.

It looks like you are genuinely running out of free states or file descriptors.

Had you applied any tuning that may have been lost in the upgrade ?

How many packets and sessions is this host meant to be handling - and what sort of traffic ?

--
Alex

On Thu, Jul 24, 2008 at 01:59:23AM -0400, Robert Jameson wrote:
> Hello Everyone,
> 
> Recently I upgraded to freebsd 7.0-p3 from 7.0-p2, once i upgraded i began
> to have problems with my network, nothing has changed configuration wise,
> let me show you guy's an example.
> 
> (12:46 AM):(root at cube)/$ ping google.com
> PING google.com (72.14.207.99): 56 data bytes
> 64 bytes from 72.14.207.99: icmp_seq=0 ttl=240 time=64.713 ms
> ^C
> --- google.com ping statistics ---
> 1 packets transmitted, 1 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 64.713/64.713/64.713/0.000 ms
> 
> (12:46 AM):(root at cube)/$ ping google.com
> PING google.com (72.14.207.99): 56 data bytes
> 64 bytes from 72.14.207.99: icmp_seq=0 ttl=240 time=73.814 ms
> 64 bytes from 72.14.207.99: icmp_seq=1 ttl=240 time=64.943 ms
> ^C
> --- google.com ping statistics ---
> 2 packets transmitted, 2 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 64.943/69.379/73.814/4.435 ms
> 
> (12:46 AM):(root at cube)/$ ping google.com
> PING google.com (72.14.207.99): 56 data bytes
> ping: sendto: Operation not permitted
> ^C
> --- google.com ping statistics ---
> 1 packets transmitted, 0 packets received, 100.0% packet loss
> (12:46 AM):(root at cube)/$
> 
> 
> As you can see above, I issued the ping command (4) times waiting for output
> and then doing CTRL+C to interrupt the commands quickly and send them again
> on the 4th try i did not intterupt it and received the operation not
> permitted. hitting ctrl+c on this error I can type ping again and it will
> work correctly.
> 
> 
> I have the same problem with almost every network command, wget, curl,
> fetch, lynx, ssh, nslookup, host etc.
> 
> 
> 
> This appears to be an issue with the network.
> 
> I have attached my rc.conf and sysctl.conf and pf.conf please let me know if
> any other information is required.
> 
> 
> Errors from /var/log/console.log:
> 
> Jul 18 21:10:02 cube kernel: Jul 18 21:10:02 cube named[908]: socket: too
> many open file descriptors
> Jul 19 00:30:13 cube kernel: Jul 19 00:30:13 cube named[9748]: socket: too
> many open file descriptors
> Jul 19 00:30:54 cube kernel: Jul 19 00:30:14 cube last message repeated 28
> times
> 
> 
> 
> Initially I figured this problem was bind related and since it has been a
> planned project for the past few months to switch to djbdns, I took the time
> to switch to djbdns, so bind is no longer running.
> 
> I was also receiving this in /var/log/messages:
> 
> Jul 20 22:15:39 cube kernel: Limiting open port RST response from 318 to 200
> packets/sec
> Jul 20 22:15:40 cube kernel: Limiting open port RST response from 624 to 200
> packets/sec
> Jul 20 22:15:42 cube kernel: Limiting open port RST response from 213 to 200
> packets/sec
> Jul 20 22:15:50 cube kernel: Limiting open port RST response from 439 to 200
> packets/sec
> Jul 20 22:15:51 cube kernel: Limiting open port RST response from 673 to 200
> packets/sec
> Jul 20 22:15:52 cube kernel: Limiting open port RST response from 730 to 200
> packets/sec
> Jul 20 22:15:53 cube kernel: Limiting open port RST response from 307 to 200
> packets/sec
> Jul 20 22:16:02 cube kernel: Limiting open port RST response from 435 to 200
> packets/sec
> Jul 20 22:16:03 cube kernel: Limiting open port RST response from 730 to 200
> packets/sec
> Jul 20 22:16:04 cube kernel: Limiting open port RST response from 287 to 200
> packets/sec
> Jul 20 22:16:13 cube kernel: Limiting open port RST response from 519 to 200
> packets/sec
> Jul 20 22:16:14 cube kernel: Limiting open port RST response from 740 to 200
> packets/sec
> Jul 20 22:16:15 cube kernel: Limiting open port RST response from 258 to 200
> packets/sec
> Jul 20 22:16:24 cube kernel: Limiting open port RST response from 407 to 200
> packets/sec
> Jul 20 22:16:25 cube kernel: Limiting open port RST response from 660 to 200
> packets/sec
> 
> After spending some time on Google i came up with:
> 
> /etc/sysctl.conf
> net.inet.icmp.icmplim=2000
> 
> I know it seems abit high, but i kept adjusting until the error went away.
> (not really fixing the problem?)
> 
> If your mail client or the mailing list prevents you from seeing the
> attached
> You can view them here:
>  http://rj.dawnshosting.com/fbsd_ml/
> 
> 
> PS: While running tcpdump I see this
> 
> tcpdump -i fxp0
> 
> Neither one of these ip's exist on my system is my cable company doing
> something wrong?
> 
> 
> 01:47:12.135929 arp who-has 64.253.3.161.dyn-cm-pool73.pool.hargray.net tell
> 64.253.3.1.dyn-cm-pool73.pool.hargray.net
> 01:47:12.155931 arp who-has 216.16.218.141.dyn-cm-pool46.pool.hargray.nettell
> 216.16.218.1.dyn-cm-pool46.pool.hargray.net
> 01:47:12.196000 arp who-has 181.131.216.67.181.static.hargray.net tell
> 1.131.216.67.1.static.hargray.net
> 
> 
> tcpdump -i fxp0 | grep ICMP:
> 
> Is this an attack?
> 
> 01:55:41.231722 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37084, length 64
> 01:55:42.232794 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37085, length 64
> 01:55:43.285913 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37086, length 64
> 01:55:44.286340 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37087, length 64
> 01:55:45.287380 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37088, length 64
> 01:55:46.345843 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37089, length 64
> 01:55:47.346685 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37090, length 64
> 01:55:48.347366 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37091, length 64
> 01:55:49.348370 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37092, length 64
> 01:55:50.360130 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37093, length 64
> 01:55:51.596916 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37094, length 64
> 01:55:52.597659 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37095, length 64
> 01:55:53.640120 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37096, length 64
> 01:55:54.735275 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37097, length 64
> 01:55:55.735568 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37098, length 64
> 01:55:56.745012 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37099, length 64
> 01:55:57.835442 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37100, length 64
> 01:55:58.920583 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37101, length 64
> 01:56:00.022747 IP cube.dawnshosting.com > purple.haze.bluntroll.in: ICMP
> echo request, id 22055, seq 37102, length 64
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20080724/d3ae2efe/attachment.pgp