Help:: Listen queue overflow killing servers

Paul Macdonald paul at
Fri Jul 26 22:02:46 UTC 2019

On 26/07/2019 22:18, James B. Byrne via freebsd-questions wrote:
>>>>>> On 7/26/19 4:58 AM, Paul Macdonald via freebsd-questions wrote:
>>>>>>> Over the past few months i've seen several boxes (4 or 5) become
>>>>>>> unresponsive as a result of a Listen queue overflow state.
> Since upgrading our hosts to 12.0 we have experienced many 'lockouts'
> of both bhyve vm guests and jails, all running on zfs.  There are
> known issues with Bhyve guests getting into a deadlock state waiting
> on zio or encountering memory exhaustion in releases after
> FreeBSD-11.1. This has multiple causes:
> We have worked around this by restricting max ARC to the net of system
> memory minus all vm allocations minus 4GB.  We also noted that as the
> capacity of zfs pools approached 80% the deadlock is encountered more
> frequently.

The ARC on the system that went down today was unbounded, and stupidly 
large  (60GB) but we had 30GB free so i didn't see harm, although i've 
now restricted.

(now: Max of 10GB, currently 4GB wired, 118GB free.)

We're totally aware of keeping ZFS below 80% utilisation, we've seen 
major performance cliffs if thats reached in the past.

In this case we're at 2% utilisation on a new (<1 month) 1TB NVMe on PCI-e

No bhyve, just 1 jail.

This was a high spec box under no significant load, but now the fifth 
box in a few months thats crumpled under listen queue overflows


Paul Macdonald
Web and video hosting
t: 0131 5548070
m: 07970339546
e: paul at
40 Maritime Street

Virtual Servers from £50.00pm
High specification Dedicated Servers from £150.00pm


More information about the freebsd-questions mailing list