Help:: Listen queue overflow killing servers
mafsys1234 at gmail.com
Fri Jul 26 13:25:20 UTC 2019
On 26/07/2019 11:08 pm, Paul Macdonald via freebsd-questions wrote:
> On 26/07/2019 13:52, Robert Schulze wrote:
>> Am 26.07.19 um 13:58 schrieb Paul Macdonald via freebsd-questions:
>> I think, these processes waiting for disk i/o are actually your problem.
>> Since they cannot answer further requests, they run into the listen
>> queue overflow.
>> You should check the processes with procstat:
>> list kernel threads:
>> # procstat -kk <PID>
>> list open files:
>> # procstat -f <PID>
> One of the things we do (whihc may be bad) is to log to a single file
> ( e.g all.sites.log, this doesn;t seem to cause problems in general ,
> but i can see how if there's X child processes then they may all need
> write locks)
Unless it's hammering the log and there's locks on it.
> Is that a really bad idea? ( Often handy to have one file for
> differnet vhosts, but maybe that needs a rethink)
Syslog works this way.
> In this case the drive is NVMe, and there's actually only a handful
> of sites, (Other servers have several hundreds of sites, much busy but
> don;t display the issue)
> In answer to some of the other suggestions, its not actually under
> high load ( 5000 lines in the apache log for the whole day), and
> system has 16C/32T, 128GB RAM
> ZFS is using a bunch of RAM as i've not limited the ARC, but there's
> 27GB free currently.
> I guess i actually have 2 questions
> 1) why are the queues filling up (i'll revert to seperate logs to see
> if that helps, although the issue is sporadic, and first time on this
Most likely (given limited information) because the processes are disk
bound doing something. They're not able to service the queue, thus
leaving a growing queue for the system.
> 2) Once the queues are over limit, is this actually unresolvable other
> than a hard reboot?
No, you can keep running, you'll just get worse and worse... :-(
You need to track down what the processes are doing, as per others'
suggestions, and then work back from there.
>> with kind regards,
>> Robert Schulze
>> freebsd-questions at freebsd.org mailing list
>> To unsubscribe, send any mail to
>> "freebsd-questions-unsubscribe at freebsd.org"
More information about the freebsd-questions