Help:: Listen queue overflow killing servers

[Christos Chatzaras]

Do you have "sendfile" enabled in Apache? If yes disable it and see if the issue is resolved.


> On 26 Jul 2019, at 14:58, Paul Macdonald via freebsd-questions <freebsd-questions at freebsd.org> wrote:
> 
> 
> Hi,
> 
> Over the past few months i've seen several boxes (4 or 5) become unresponsive as a result of a Listen queue overflow state.
> 
> Processes stack up, none are killable, all these are within jails and neither the jail can be stopped nor the server rebooted (without a power cycle).
> 
> All are on ZFS and are std apache/php/mysql servers with nothing too exotic.
> 
> All on 12.0-RELEASE, i've only started seeing these issues recently, but it feels like more and more.
> 
> /var/log/messages shows tyically;
> 
>     kernel: sonewconn: pcb 0xfffff813395e3d58: Listen queue overflow: 193 already in queue awaiting acceptance (83 occurrences)
> 
> netstat -Lan  shows
> 
> tcp4 193/0/128                          x.x.x.x.443
> tcp4  193/0/128                          x.x.x.x.80
> 
> connections cannot be killed with tcpdrop ( except ssh which can!)
> 
> All processes seem to be in Disk State ( many many apache processes but others getting stuck too)
> 
> www      60089    0.0 0.1  196588   78328  -  DJ   21:07         1:19.54 /usr/local/sbin/httpd -DNOHTTPACCEPT
> ..<snoip>
> 
> www      93713    0.0 0.0  183576   33164  -  DJ   23:57         0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT
> 
> but no zombies..
> 
> last pid: 24773;  load averages:  0.00,  0.00, 0.00                       up 52+11:41:09  11:48:02
> 918 processes: 1 running, 917 sleeping
> CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
> Mem: 107M Active, 3729M Inact, 93G Wired, 27G Free
> ARC: 79G Total, 54G MFU, 23G MRU, 243M Anon, 710M Header, 1615M Other
>      73G Compressed, 191G Uncompressed, 2.60:1 Ratio
> Swap: 4096M Total, 4096M Free
> 
> 
> I'd appreciate any advice as at present it looks like my only option is to hard power cycle these
> 
> many thanks
> 
> Paul.
> 
> 
> 
> -- 
> -------------------------
> Paul Macdonald
> IFDNRG Ltd
> Web and video hosting
> -------------------------
> t: 0131 5548070
> m: 07970339546
> e: paul at ifdnrg.com
> w: http://www.ifdnrg.com
> -------------------------
> IFDNRG
> 40 Maritime Street
> Edinburgh
> EH6 6SA
> ----------------------------------------------------
> 
> Virtual Servers from £50.00pm
> High specification Dedicated Servers from £150.00pm
> 
> ----------------------------------------------------
> 
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org"