Help:: Listen queue overflow killing servers

MJ mafsys1234 at
Fri Jul 26 13:25:20 UTC 2019

On 26/07/2019 11:08 pm, Paul Macdonald via freebsd-questions wrote:
> On 26/07/2019 13:52, Robert Schulze wrote:
>> Hi,
>> Am 26.07.19 um 13:58 schrieb Paul Macdonald via freebsd-questions:
>> I think, these processes waiting for disk i/o are actually your problem.
>> Since they cannot answer further requests, they run into the listen
>> queue overflow.
>> You should check the processes with procstat:
>> list kernel threads:
>> # procstat -kk <PID>
>> list open files:
>> # procstat -f <PID>
> One of the things we do (whihc may be bad)  is to log to a single file 
> ( e.g all.sites.log, this doesn;t seem to cause problems in general , 
> but i can see how if there's X child processes then they may all need 
> write locks)
Unless it's hammering the log and there's locks on it.

> Is that a really bad idea? ( Often handy to have one file for 
> differnet vhosts, but maybe that needs a rethink)

Syslog works this way.

> In this case the drive is NVMe, and there's actually  only a handful 
> of sites, (Other servers have several hundreds of sites, much busy but 
> don;t display  the issue)
> In answer to some of the other suggestions, its not actually under 
> high load ( 5000 lines in the apache log for the whole day), and 
> system has 16C/32T,  128GB RAM
> ZFS is using a bunch of RAM as i've not limited the ARC, but there's 
> 27GB free currently.
> I guess i actually have 2 questions
> 1) why are the queues filling up (i'll  revert to seperate logs to see 
> if that helps, although the issue is sporadic, and first time on this 
> box)
Most likely (given limited information) because the processes are disk 
bound doing something. They're not able to service the queue, thus 
leaving a growing queue for the system.

> 2) Once the queues are over limit, is this actually unresolvable other 
> than a hard reboot?
No, you can keep running, you'll just get worse and worse... :-(

You need to track down what the processes are doing, as per others' 
suggestions, and then work back from there.

>> with kind regards,
>> Robert Schulze
>> _______________________________________________
>> freebsd-questions at mailing list
>> To unsubscribe, send any mail to 
>> "freebsd-questions-unsubscribe at"

More information about the freebsd-questions mailing list