filling up UDP socket buffers like mad
Michael W. Lucas
mwlucas at blackhelicopters.org
Mon Mar 27 14:51:05 UTC 2006
On Fri, Mar 24, 2006 at 06:03:47PM -0500, Charles Swiger wrote:
> On Mar 24, 2006, at 4:17 PM, Michael W. Lucas wrote:
> >Running FreeBSD 6.1-PRERELEASE as a DNS, dhcp, and syslog server.
> >
> >I'm having trouble with DNS, DHCP, and syslogd locking up, and I think
> >I've found what they all share in common.
> >
> >During the lockups, the box starts dropping UDP due to full socket
> >buffers. I have a dumb little script to capture the rate of drops
> >over 5 seconds, and it's about 45 a second.
> >
> >168725 dropped due to full socket buffers
> >168958 dropped due to full socket buffers
>
> There is generally a cause behind the socket buffers filling up,
> whether that is some form of livelock due to an OS problem or a
> misconfiguration with a firewall/dummynet setup. You could look at
> the output of "netstat -a(n)" for insight as to where the packets are
> being queued up, but "netstat -s" would be useful to show to us as well.
Thanks. I think you've shown me how to find the problem:
# netstat -na
...
udp4 0 0 127.0.0.1.57058 127.0.0.1.53
udp4 0 0 127.0.0.1.61259 127.0.0.1.53
udp4 0 0 127.0.0.1.54240 127.0.0.1.53
udp4 0 0 127.0.0.1.52997 127.0.0.1.53
udp4 0 0 *.67 *.*
udp4 43414 0 *.514 *.*
udp4 0 0 *.49661 *.*
...
We have no firewall on this machine; it's buried behind three layers
of firewall.
I've tried running syslogd in debug mode, but not found anything
particularly useful yet. Syslogd is now set to restart every 15
minutes, and run in debug mode, so hopefully the next time this
happens I'll have the debugging output. The problem happens even
within fifteen minutes, but because of my timeouts nobody notices.
I'm attaching the output of netstat -na and netstat -s for general
informative purposes; if anyone has any further suggestions, I'm all
ears.
Thanks,
==ml
--
Michael W. Lucas mwlucas at FreeBSD.org, mwlucas at BlackHelicopters.org
http://www.BlackHelicopters.org/~mwlucas/
"The cloak of anonymity protects me from the nuisance of caring." -Non Sequitur
-------------- next part --------------
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 96 0 10.184.1.11.53 10.184.1.8.49838 ESTABLISHED
tcp4 104 0 10.184.1.11.53 10.184.1.8.61719 ESTABLISHED
tcp4 0 0 127.0.0.1.6011 *.* LISTEN
tcp4 0 48 10.184.1.11.22 10.184.184.108.52392 ESTABLISHED
tcp4 0 0 127.0.0.1.6010 *.* LISTEN
tcp4 0 0 10.184.1.11.22 10.184.184.108.64567 ESTABLISHED
tcp4 0 0 *.199 *.* LISTEN
tcp4 0 0 *.80 *.* LISTEN
tcp4 0 0 127.0.0.1.25 *.* LISTEN
tcp4 0 0 *.22 *.* LISTEN
tcp4 0 0 127.0.0.1.953 *.* LISTEN
tcp4 0 0 127.0.0.1.53 *.* LISTEN
tcp4 0 0 10.184.3.248.53 *.* LISTEN
tcp4 0 0 10.184.1.11.53 *.* LISTEN
udp4 0 0 127.0.0.1.59578 127.0.0.1.53
udp4 0 0 127.0.0.1.57058 127.0.0.1.53
udp4 0 0 127.0.0.1.61259 127.0.0.1.53
udp4 0 0 127.0.0.1.54240 127.0.0.1.53
udp4 0 0 127.0.0.1.52997 127.0.0.1.53
udp4 0 0 *.67 *.*
udp4 43414 0 *.514 *.*
udp4 0 0 *.49661 *.*
udp4 0 0 *.52767 *.*
udp4 0 0 *.63357 *.*
udp4 0 0 127.0.0.1.58915 127.0.0.1.5680
udp4 0 0 127.0.0.1.62275 127.0.0.1.5679
udp4 0 0 127.0.0.1.57431 127.0.0.1.5678
udp4 0 0 *.63077 *.*
udp4 0 0 *.161 *.*
udp4 0 0 *.57589 *.*
udp4 0 0 *.5681 *.*
udp4 0 0 127.0.0.1.5680 *.*
udp4 0 0 127.0.0.1.5679 *.*
udp4 0 0 127.0.0.1.5678 *.*
udp4 0 0 *.64486 *.*
udp4 0 0 *.53558 *.*
udp4 0 0 127.0.0.1.123 *.*
udp4 0 0 10.184.3.248.123 *.*
udp4 0 0 10.184.1.11.123 *.*
udp4 0 0 *.123 *.*
udp4 909 0 *.58684 *.*
udp4 4129 0 127.0.0.1.53 *.*
udp4 278 0 10.184.3.248.53 *.*
udp4 8436 0 10.184.1.11.53 *.*
icm4 0 0 *.* *.*
Active UNIX domain sockets
Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr
c332d690 stream 0 0 c31c4bb0 0 0 0 /tmp/ssh-GceIKkNhTt/agent.1161
c332d71c stream 0 0 0 c332d460 0 0
c332d460 stream 0 0 0 c332d71c 0 0
c332dc08 stream 0 0 c31aa770 0 0 0 /tmp/ssh-YFEQdkNEm5/agent.1148
c332d8c0 stream 0 0 0 c332de38 0 0
c332de38 stream 0 0 0 c332d8c0 0 0
c332d08c stream 0 0 c3667330 0 0 0 /var/run/services.softflowd.ctl
c332d578 stream 0 0 c3667660 0 0 0 /var/run/mpls.softflowd.ctl
c332d604 stream 0 0 c3667990 0 0 0 /var/run/anx.softflowd.ctl
c332e000 stream 0 0 c3146cc0 0 0 0 /var/run/devd.pipe
c332d834 dgram 0 0 0 c332dc94 0 c332d7a8
c332d7a8 dgram 0 0 0 c332dc94 0 c332d118
c332d118 dgram 0 0 0 c332dc94 0 c332db7c
c332d2bc dgram 0 0 0 c332d348 0 0
c332db7c dgram 0 0 0 c332dc94 0 0
c332d348 dgram 4048 0 c32a5660 0 c332d2bc 0 /var/named/var/run/log
c332dd20 dgram 0 0 c32a5770 0 0 0 /var/run/log
c332dc94 dgram 4048 0 c32a5880 0 c332d834 0 /var/run/logpriv
c332d000 dgram 0 0 c32a5990 0 0 0 /var/run/log
c332dec4 dgram 0 0 0 0 0 c332d7a8
c332d4ec dgram 0 0 0 0 0 c332d8c0
c332d9d8 dgram 0 0 0 0 0 0
c332da64 dgram 0 0 0 0 0 c332db7c
tcp:
1986 packets sent
1254 data packets (124262 bytes)
0 data packets (0 bytes) retransmitted
0 data packets unnecessarily retransmitted
0 resends initiated by MTU discovery
486 ack-only packets (35 delayed)
0 URG only packets
0 window probe packets
0 window update packets
246 control packets
2428 packets received
1587 acks (for 124643 bytes)
23 duplicate acks
0 acks for unsent data
1142 packets (79497 bytes) received in-sequence
0 completely duplicate packets (0 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
0 out-of-order packets (0 bytes)
0 packets (0 bytes) of data after window
0 window probes
8 window update packets
0 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
25 connection requests
208 connection accepts
2 bad connection attempts
0 listen queue overflows
0 ignored RSTs in the windows
225 connections established (including accepts)
232 connections closed (including 0 drops)
26 connections updated cached RTT on close
26 connections updated cached RTT variance on close
0 connections updated cached ssthresh on close
8 embryonic connections dropped
1392 segments updated rtt (of 1373 attempts)
0 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
0 connections dropped by persist timeout
0 keepalive timeouts
0 keepalive probes sent
0 connections dropped by keepalive
187 correct ACK header predictions
525 correct data packet header predictions
208 syncache entries added
0 retransmitted
0 dupsyn
0 dropped
208 completed
0 bucket overflow
0 cache overflow
0 reset
0 stale
0 aborted
0 badack
0 unreach
0 zone failures
0 cookies sent
0 cookies received
0 SACK recovery episodes
0 segment rexmits in SACK recovery episodes
0 byte rexmits in SACK recovery episodes
0 SACK options (SACK blocks) received
0 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
udp:
550908 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
0 with no checksum
33451 dropped due to no socket
63890 broadcast/multicast datagrams dropped due to no socket
63147 dropped due to full socket buffers
0 not for hashed pcb
390420 delivered
276481 datagrams output
ip:
560670 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with ip length > max ip packet size
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 packets reassembled ok
554968 packets for this host
42 packets for unknown/unsupported protocol
0 packets forwarded (0 packets fast forwarded)
3059 packets not forwardable
2601 packets received for unknown multicast group
0 redirects sent
312928 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 tunneling packets that can't find gif
0 datagrams with bad address in header
icmp:
33294 calls to icmp_error
0 errors not generated in response to an icmp message
Output histogram:
echo reply: 1088
destination unreachable: 33294
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
0 multicast echo requests ignored
0 multicast timestamp requests ignored
Input histogram:
echo reply: 16
destination unreachable: 261
routing redirect: 309
echo: 1088
1088 message responses generated
0 invalid return addresses
0 no return routes
ICMP address mask responses are disabled
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent
More information about the freebsd-questions
mailing list