Troubles with Kibana

From: Andrea Venturoli <ml_at_netfence.it>
Date: Sat, 03 Feb 2024 15:24:48 UTC
Hello.

I've got a 13.2p9/amd64 server where I have a jail dedicated to 
ElasticSearch+Kibana and I'm having troubles starting and stopping the 
latter.

> # pkg info | grep -E "(elastic|kibana)"
> elasticsearch8-8.11.3          Distributed, RESTful search and analytics engine
> kibana8-8.11.3                 Browser based analytics and search interface to ElasticSearch
> py39-elasticsearch-7.17.9      Official Python low-level client for Elasticsearch
> py39-elasticsearch-dsl-7.3.0   High level Python client for Elasticsearch

> # cat /etc/rc.conf
> #cron_enable="NO"
> kibana_enable="YES"
> kibana_syslog_output_enable="YES"
> elasticsearch_enable="YES"





First problem

When I start the jail (with "ezjail-admin start") Elastic starts, but 
Kibana does not.
syslog shows:
> [2024-02-03T16:14:21.861+01:00][INFO ][root] Kibana is starting
> [2024-02-03T16:14:21.939+01:00][INFO ][root] Kibana is shutting down
> [2024-02-03T16:14:21.939+01:00][FATAL][root] Reason: EACCES: permission denied, open '/var/run/kibana.pid'
> Error: EACCES: permission denied, open '/var/run/kibana.pid'
> 
>  FATAL  Error: EACCES: permission denied, open '/var/run/kibana.pid'

Same happens if I try "service kibana start" afterwards from inside the 
jail.
I can get around this with:
> # touch /var/run/kibana.pid
> # chown www /var/run/kibana.pid
> # service kibana start

However this must be done manually everytime I need to restart the 
server or jail.





Second problem

When I try to stop Kibana, it just hangs indefinitely:
>  # service kibana stop
> Stopping kibana.
> Waiting for PIDS: 28179

syslog reports:
> [2024-02-03T16:20:22.309+01:00][INFO ][root] SIGTERM received - initiating shutdown
> [2024-02-03T16:20:22.309+01:00][INFO ][root] Kibana is shutting down
> [2024-02-03T16:20:22.311+01:00][INFO ][plugins-system.standard] Stopping all plugins.
> [2024-02-03T16:20:22.314+01:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Monitoring stats collection is stopped
> [2024-02-03T16:20:22.323+01:00][INFO ][plugins-system.standard] All plugins stopped.
> [2024-02-03T16:20:22.324+01:00][WARN ][environment] Detected an unhandled Promise rejection: Error: EACCES: permission denied, unlink '/var/run/kibana.pid'
>     at unlinkSync (node:fs:1883:3)
>     at process.<anonymous> (/usr/local/www/kibana8/node_modules/@kbn/core-environment-server-internal/src/write_pid_file.js:49:24)
>     at process.<anonymous> (/usr/local/www/kibana8/node_modules/lodash/before.js:31:21)
>     at Object.onceWrapper (node:events:632:26)
>     at process.emit (node:events:529:35)
>     at process.emit (/usr/local/www/kibana8/node_modules/source-map-support/source-map-support.js:516:21)
>     at process.exit (node:internal/process/per_thread:192:15)
>     at Root.onRootShutdown [as onShutdown] (/usr/local/www/kibana8/node_modules/@kbn/core-root-server-internal/src/bootstrap.js:136:11)
>     at Root.shutdown (/usr/local/www/kibana8/node_modules/@kbn/core-root-server-internal/src/root/index.js:96:12)
>     at processTicksAndRejections (node:internal/process/task_queues:95:5)
> [2024-02-03T16:20:23.026+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:20:26.024+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:20:29.024+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:20:32.023+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:20:35.022+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:20:38.021+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:20:41.021+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:20:44.021+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:20:47.022+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:20:50.022+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:20:53.023+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:20:56.022+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:20:59.021+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:21:02.021+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:21:05.021+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:21:08.022+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:21:11.023+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:21:14.022+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:21:17.021+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections
> [2024-02-03T16:21:20.020+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections

This goes on until I kill -9 the process.

This is particularly problematic in case power goes missing and the UPS 
tries to shutdown the server, which will hang, instead.



Any hint?

  bye & Thanks
	av.