Troubles with Kibana
- Reply: Miroslav Lachman : "Re: Troubles with Kibana"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Sat, 03 Feb 2024 15:24:48 UTC
Hello. I've got a 13.2p9/amd64 server where I have a jail dedicated to ElasticSearch+Kibana and I'm having troubles starting and stopping the latter. > # pkg info | grep -E "(elastic|kibana)" > elasticsearch8-8.11.3 Distributed, RESTful search and analytics engine > kibana8-8.11.3 Browser based analytics and search interface to ElasticSearch > py39-elasticsearch-7.17.9 Official Python low-level client for Elasticsearch > py39-elasticsearch-dsl-7.3.0 High level Python client for Elasticsearch > # cat /etc/rc.conf > #cron_enable="NO" > kibana_enable="YES" > kibana_syslog_output_enable="YES" > elasticsearch_enable="YES" First problem When I start the jail (with "ezjail-admin start") Elastic starts, but Kibana does not. syslog shows: > [2024-02-03T16:14:21.861+01:00][INFO ][root] Kibana is starting > [2024-02-03T16:14:21.939+01:00][INFO ][root] Kibana is shutting down > [2024-02-03T16:14:21.939+01:00][FATAL][root] Reason: EACCES: permission denied, open '/var/run/kibana.pid' > Error: EACCES: permission denied, open '/var/run/kibana.pid' > > FATAL Error: EACCES: permission denied, open '/var/run/kibana.pid' Same happens if I try "service kibana start" afterwards from inside the jail. I can get around this with: > # touch /var/run/kibana.pid > # chown www /var/run/kibana.pid > # service kibana start However this must be done manually everytime I need to restart the server or jail. Second problem When I try to stop Kibana, it just hangs indefinitely: > # service kibana stop > Stopping kibana. > Waiting for PIDS: 28179 syslog reports: > [2024-02-03T16:20:22.309+01:00][INFO ][root] SIGTERM received - initiating shutdown > [2024-02-03T16:20:22.309+01:00][INFO ][root] Kibana is shutting down > [2024-02-03T16:20:22.311+01:00][INFO ][plugins-system.standard] Stopping all plugins. > [2024-02-03T16:20:22.314+01:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Monitoring stats collection is stopped > [2024-02-03T16:20:22.323+01:00][INFO ][plugins-system.standard] All plugins stopped. > [2024-02-03T16:20:22.324+01:00][WARN ][environment] Detected an unhandled Promise rejection: Error: EACCES: permission denied, unlink '/var/run/kibana.pid' > at unlinkSync (node:fs:1883:3) > at process.<anonymous> (/usr/local/www/kibana8/node_modules/@kbn/core-environment-server-internal/src/write_pid_file.js:49:24) > at process.<anonymous> (/usr/local/www/kibana8/node_modules/lodash/before.js:31:21) > at Object.onceWrapper (node:events:632:26) > at process.emit (node:events:529:35) > at process.emit (/usr/local/www/kibana8/node_modules/source-map-support/source-map-support.js:516:21) > at process.exit (node:internal/process/per_thread:192:15) > at Root.onRootShutdown [as onShutdown] (/usr/local/www/kibana8/node_modules/@kbn/core-root-server-internal/src/bootstrap.js:136:11) > at Root.shutdown (/usr/local/www/kibana8/node_modules/@kbn/core-root-server-internal/src/root/index.js:96:12) > at processTicksAndRejections (node:internal/process/task_queues:95:5) > [2024-02-03T16:20:23.026+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:20:26.024+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:20:29.024+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:20:32.023+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:20:35.022+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:20:38.021+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:20:41.021+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:20:44.021+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:20:47.022+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:20:50.022+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:20:53.023+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:20:56.022+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:20:59.021+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:21:02.021+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:21:05.021+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > NoLivingConnectionsError: There are no living connections > [2024-02-03T16:21:08.022+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:21:11.023+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:21:14.022+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:21:17.021+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections > [2024-02-03T16:21:20.020+01:00][ERROR][plugins.taskManager] Failed to poll for work: NoLivingConnectionsError: There are no living connections This goes on until I kill -9 the process. This is particularly problematic in case power goes missing and the UPS tries to shutdown the server, which will hang, instead. Any hint? bye & Thanks av.