Help:: Listen queue overflow killing servers

[MJ]

On 26/07/2019 11:08 pm, Paul Macdonald via freebsd-questions wrote:
>
> On 26/07/2019 13:52, Robert Schulze wrote:
>> Hi,
>>
>> Am 26.07.19 um 13:58 schrieb Paul Macdonald via freebsd-questions:
>> I think, these processes waiting for disk i/o are actually your problem.
>> Since they cannot answer further requests, they run into the listen
>> queue overflow.
>>
>> You should check the processes with procstat:
>>
>> list kernel threads:
>> # procstat -kk <PID>
>>
>> list open files:
>> # procstat -f <PID>
>
>
> One of the things we do (whihc may be bad)  is to log to a single file 
> ( e.g all.sites.log, this doesn;t seem to cause problems in general , 
> but i can see how if there's X child processes then they may all need 
> write locks)
>
Unless it's hammering the log and there's locks on it.


> Is that a really bad idea? ( Often handy to have one file for 
> differnet vhosts, but maybe that needs a rethink)

Syslog works this way.


>
> In this case the drive is NVMe, and there's actually  only a handful 
> of sites, (Other servers have several hundreds of sites, much busy but 
> don;t display  the issue)
>
> In answer to some of the other suggestions, its not actually under 
> high load ( 5000 lines in the apache log for the whole day), and 
> system has 16C/32T,  128GB RAM
>
> ZFS is using a bunch of RAM as i've not limited the ARC, but there's 
> 27GB free currently.
>
> I guess i actually have 2 questions
>
> 1) why are the queues filling up (i'll  revert to seperate logs to see 
> if that helps, although the issue is sporadic, and first time on this 
> box)
>
Most likely (given limited information) because the processes are disk 
bound doing something. They're not able to service the queue, thus 
leaving a growing queue for the system.


> 2) Once the queues are over limit, is this actually unresolvable other 
> than a hard reboot?
>
No, you can keep running, you'll just get worse and worse... :-(

You need to track down what the processes are doing, as per others' 
suggestions, and then work back from there.


>
>
>> with kind regards,
>> Robert Schulze
>> _______________________________________________
>> freebsd-questions at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-questions
>> To unsubscribe, send any mail to 
>> "freebsd-questions-unsubscribe at freebsd.org"
>>