Help:: Listen queue overflow killing servers

[Robert Schulze]

Hi,

Am 26.07.19 um 13:58 schrieb Paul Macdonald via freebsd-questions:
> Over the past few months i've seen several boxes (4 or 5) become
> unresponsive as a result of a Listen queue overflow state.
> 
> Processes stack up, none are killable, all these are within jails and
> neither the jail can be stopped nor the server rebooted (without a power
> cycle).
> 
> All are on ZFS and are std apache/php/mysql servers with nothing too
> exotic.
> 
> All on 12.0-RELEASE, i've only started seeing these issues recently, but
> it feels like more and more.
> 
> /var/log/messages shows tyically;
> 
>     kernel: sonewconn: pcb 0xfffff813395e3d58: Listen queue overflow:
> 193 already in queue awaiting acceptance (83 occurrences)
> 
> netstat -Lan  shows
> 
> tcp4 193/0/128                          x.x.x.x.443
> tcp4  193/0/128                          x.x.x.x.80
> 
> connections cannot be killed with tcpdrop ( except ssh which can!)
> 
> All processes seem to be in Disk State ( many many apache processes but
> others getting stuck too)
> 
> www      60089    0.0 0.1  196588   78328  -  DJ   21:07         1:19.54
> /usr/local/sbin/httpd -DNOHTTPACCEPT
> ..<snoip>
> 
> www      93713    0.0 0.0  183576   33164  -  DJ   23:57         0:00.01
> /usr/local/sbin/httpd -DNOHTTPACCEPT
> 

I think, these processes waiting for disk i/o are actually your problem.
Since they cannot answer further requests, they run into the listen
queue overflow.

You should check the processes with procstat:

list kernel threads:
# procstat -kk <PID>

list open files:
# procstat -f <PID>

with kind regards,
Robert Schulze