Help:: Listen queue overflow killing servers

[Paul Macdonald]

On 26/07/2019 22:18, James B. Byrne via freebsd-questions wrote:
>
>>>>>> On 7/26/19 4:58 AM, Paul Macdonald via freebsd-questions wrote:
>>>>>>> Over the past few months i've seen several boxes (4 or 5) become
>>>>>>> unresponsive as a result of a Listen queue overflow state.
> Since upgrading our hosts to 12.0 we have experienced many 'lockouts'
> of both bhyve vm guests and jails, all running on zfs.  There are
> known issues with Bhyve guests getting into a deadlock state waiting
> on zio or encountering memory exhaustion in releases after
> FreeBSD-11.1. This has multiple causes:
>
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231117
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594
>
> We have worked around this by restricting max ARC to the net of system
> memory minus all vm allocations minus 4GB.  We also noted that as the
> capacity of zfs pools approached 80% the deadlock is encountered more
> frequently.

The ARC on the system that went down today was unbounded, and stupidly 
large  (60GB) but we had 30GB free so i didn't see harm, although i've 
now restricted.

(now: Max of 10GB, currently 4GB wired, 118GB free.)

We're totally aware of keeping ZFS below 80% utilisation, we've seen 
major performance cliffs if thats reached in the past.

In this case we're at 2% utilisation on a new (<1 month) 1TB NVMe on PCI-e

No bhyve, just 1 jail.

This was a high spec box under no significant load, but now the fifth 
box in a few months thats crumpled under listen queue overflows

Paul






-- 
-------------------------
Paul Macdonald
IFDNRG Ltd
Web and video hosting
-------------------------
t: 0131 5548070
m: 07970339546
e: paul at ifdnrg.com
w: http://www.ifdnrg.com
-------------------------
IFDNRG
40 Maritime Street
Edinburgh
EH6 6SA
----------------------------------------------------

Virtual Servers from £50.00pm
High specification Dedicated Servers from £150.00pm

----------------------------------------------------