Help:: Listen queue overflow killing servers

[Igor V. Ruzanov]

You can increase TCP queue at system level:
sysctl kern.ipc.somaxconn=65535 (its maximum value). But often its not 
enough because of application you use. So dig into application settings to 
increase the queue of TCP sockets to be handled in your system.


On Fri, 26 Jul 2019, Paul Macdonald via freebsd-questions wrote:

|
|Hi,
|
|Over the past few months i've seen several boxes (4 or 5) become unresponsive
|as a result of a Listen queue overflow state.
|
|Processes stack up, none are killable, all these are within jails and neither
|the jail can be stopped nor the server rebooted (without a power cycle).
|
|All are on ZFS and are std apache/php/mysql servers with nothing too exotic.
|
|All on 12.0-RELEASE, i've only started seeing these issues recently, but it
|feels like more and more.
|
|/var/log/messages shows tyically;
|
|    kernel: sonewconn: pcb 0xfffff813395e3d58: Listen queue overflow: 193
|already in queue awaiting acceptance (83 occurrences)
|
|netstat -Lan  shows
|
|tcp4 193/0/128                          x.x.x.x.443
|tcp4  193/0/128                          x.x.x.x.80
|
|connections cannot be killed with tcpdrop ( except ssh which can!)
|
|All processes seem to be in Disk State ( many many apache processes but others
|getting stuck too)
|
|www      60089    0.0 0.1  196588   78328  -  DJ   21:07         1:19.54
|/usr/local/sbin/httpd -DNOHTTPACCEPT
|..<snoip>
|
|www      93713    0.0 0.0  183576   33164  -  DJ   23:57         0:00.01
|/usr/local/sbin/httpd -DNOHTTPACCEPT
|
|but no zombies..
|
|last pid: 24773;  load averages:  0.00,  0.00, 0.00                       up
|52+11:41:09  11:48:02
|918 processes: 1 running, 917 sleeping
|CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
|Mem: 107M Active, 3729M Inact, 93G Wired, 27G Free
|ARC: 79G Total, 54G MFU, 23G MRU, 243M Anon, 710M Header, 1615M Other
|     73G Compressed, 191G Uncompressed, 2.60:1 Ratio
|Swap: 4096M Total, 4096M Free
|
|
|I'd appreciate any advice as at present it looks like my only option is to hard
|power cycle these
|
|many thanks
|
|Paul.
|
|
|
|-- 
|-------------------------
|Paul Macdonald
|IFDNRG Ltd
|Web and video hosting
|-------------------------
|t: 0131 5548070
|m: 07970339546
|e: paul at ifdnrg.com
|w: http://www.ifdnrg.com
|-------------------------
|IFDNRG
|40 Maritime Street
|Edinburgh
|EH6 6SA
|----------------------------------------------------
|
|Virtual Servers from £50.00pm
|High specification Dedicated Servers from £150.00pm
|
|----------------------------------------------------
|
|_______________________________________________
|freebsd-questions at freebsd.org mailing list
|https://lists.freebsd.org/mailman/listinfo/freebsd-questions
|To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org"
|