Help:: Listen queue overflow killing servers

[Paul Macdonald]

On 26/07/2019 13:52, Robert Schulze wrote:
> Hi,
>
> Am 26.07.19 um 13:58 schrieb Paul Macdonald via freebsd-questions:
> I think, these processes waiting for disk i/o are actually your problem.
> Since they cannot answer further requests, they run into the listen
> queue overflow.
>
> You should check the processes with procstat:
>
> list kernel threads:
> # procstat -kk <PID>
>
> list open files:
> # procstat -f <PID>


One of the things we do (whihc may be bad)  is to log to a single file ( 
e.g all.sites.log, this doesn;t seem to cause problems in general , but 
i can see how if there's X child processes then they may all need write 
locks)

Is that a really bad idea? ( Often handy to have one file for differnet 
vhosts, but maybe that needs a rethink)

In this case the drive is NVMe, and there's actually  only a handful of 
sites, (Other servers have several hundreds of sites, much busy but 
don;t display  the issue)

In answer to some of the other suggestions, its not actually under high 
load ( 5000 lines in the apache log for the whole day), and system has 
16C/32T,  128GB RAM

ZFS is using a bunch of RAM as i've not limited the ARC, but there's 
27GB free currently.

I guess i actually have 2 questions

1) why are the queues filling up (i'll  revert to seperate logs to see 
if that helps, although the issue is sporadic, and first time on this box)

2)  Once the queues are over limit, is this actually unresolvable other 
than a hard reboot?

I've increased the queue size via

sysctl kern.ipc.somaxconn

to 1024 for now

but netstat -Lan still shows

tcp4  0/0/128                          x.x.x.x.443
tcp4  0/0/128                          x.x.x.x.80


thanks for assiestence


Paul






> with kind regards,
> Robert Schulze
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org"
>
-- 
-------------------------
Paul Macdonald
IFDNRG Ltd
Web and video hosting
-------------------------
t: 0131 5548070
m: 07970339546
e: paul at ifdnrg.com
w: http://www.ifdnrg.com
-------------------------
IFDNRG
40 Maritime Street
Edinburgh
EH6 6SA
----------------------------------------------------

Virtual Servers from £50.00pm
High specification Dedicated Servers from £150.00pm

----------------------------------------------------