maintainer-feedback requested: [Bug 248893] sysutils/logstash7: Update to 7.9.0
bugzilla-noreply at freebsd.org
bugzilla-noreply at freebsd.org
Mon Aug 24 21:32:35 UTC 2020
Bugzilla Automation <bugzilla at FreeBSD.org> has asked freebsd-elastic (Nobody)
<elastic at FreeBSD.org> for maintainer-feedback:
Bug 248893: sysutils/logstash7: Update to 7.9.0
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248893
--- Description ---
Hi,
please find the patch attached.
Changelog
New features and enhancements
ECS support in Elasticsearch output plugin
This release is the first step toward Elastic Common Schema (ECS)
support in Logstash. With 7.9, you can configure the Elasticsearch
output plugin to manage index templates that are compatible with the
Elastic Common Schema (ECS). The ECS compatibility setting in
the Elasticsearch output plugin makes this possible.
See Compatibility with the Elastic Common Schema (ECS) in the
Elasticsearch output plugin docs for more information.
Expanded API key support
With this release, we've continued expanding support for Elasticsearch
API keys. Support for API keys in the Elasticsearch output plugin
arrived in Logstash 7.8.0. Logstash 7.9.0 introduces support for
Elasticsearch API keys in the Elasticsearch input plugin, the
Elasticsearch filter plugin, and Logstash monitoring and
management.
Check out Grant access using API keys for more information about
using API keys with Logstash and Elasticsearch. Implementation details
are in
ARM64 support (experimental)
Logstash runs on arm machines! We have tested Logstash against arm64,
and we are looking to make docker and other images available soon.
ARM artifacts are not yet supported for production, and we're offering
them as "experimental" to early adopters.
Improved support in App Search output
We replaced the deprecated Java client library for the Elastic App
Search output plugin with the Ruby client library, and expanded
integration testing. These changes provide a foundation for expanding
App Search integration and quality assurance in future releases.
Improvements to persistent queue (PQ)
We've enhanced persistent queues to better manage exceptions and error
handling which could sometimes result in a LockException when the queue
file lock was not properly released. Under some conditions, a complex
pipeline that is slower to initialize could be recreated when it was
not done initializing, causing a LockException. Implementation details
are in
These changes result in better stability of persistent queues.
Improvements to pipeline workers error handling
Worker threads were not correctly monitored for a worker loop exception
resulting in a complete logstash crash upon any exception even when
multiple pipelines are running. Now only the failed pipeline is
terminated. If pipeline reloading is enabled, you can edit the config
and have the failed pipeline reloaded. Implementation details are in
and
Performance improvement on startup and pipeline restarts
This release contains several optimizations to pipeline compilation, an
essential step of the pipeline initialization process. These changes
significantly improve startup and pipeline-restart performance for
complex pipelines. (For technical details, check out this PR:
)
From our tests in three different pipelines with eight workers each, we
have seen times decrease from 9 - 28 minutes to around 1 minute.
To aid the development of pipelines, especially the performance impact
of compilation, Logstash now reports the time taken to compile each
pipeline as a log entry such as:
[2020-08-12T14:10:29,[INFO ][logstash.javapipeline ][main] Pipeline Java ex
ecution initialization time {"seconds"=>0.7}
Performance improvements and notable issues fixed
* Support white space as a delimiter on list-type params
* Support using unix pipe as local config file
* Logging improvements
+ Display Java pipeline initialization time to help with
troubleshooting and diagnostics
+ Logging framework enhancement to allow more finetuned logging
+ Better logging after definition improvements and script routes
in log4j and
+ Improved Logstash startup logging to ensure that starting
logstash entry happens before any other log entries
* Fix: Add back pipelines queue.data and queue.capacity subdocuments
for _node/stats
* Fix: Avoid reloading pipelines that have no changes
* Fix: Removed unnecessary calls that, under some circumstances,
could cause pipeline startup issues for pipelines that were slow to
initialize
* Fix: Allow trailing newlines in config fragments to resolve an
issue in which split configs were corrupted when merged
* Fix: Resolve issue in which pipeline init fails for a slow pipeline
when monitoring is enabled
* Fix: Ignore default username when no password is set for monitoring
and management
* Refactor code refactor to launch ruby thread from ruby code instead
of java (as a workaround for jruby bug)
* Updates to dependencies
+ Update log4j dependency to 2.13.3
+ Update jruby to 9.2.12.0
Plugin releases
Rubydebug Codec - 3.1.0
* Replace stale awesome_print library with maintained fork called
amazing_print
Elasticsearch Filter - 3.9.0
* Add support to define a proxy with the proxy config option
* Added api_key support
* [DOC] Removed outdated compatibility notice
Memcached Filter - 1.1.0
* Added better exception handling
Elasticsearch Input - 4.7.0
* Added api_key support
File Input - 4.2.1
* Fix: Skip sincedb eviction if read mode completion deletes file
during flush
* Fix: Watched files performance with huge filesets
* Updated logging to include full traces in debug (and trace) levels
Imap Input - 3.1.0
* Adds an option to recursively search the message parts for
attachment and inline attachment filenames. If the save_attachments
option is set to true, the content of attachments is included in
the attachments.data field. The attachment data can then be used by
the Elasticsearch Ingest Attachment Processor Plugin
Kafka Integration - 10.4.0
* Added the input isolation_level to allow fine control of whether to
return transactional messages
* Added the input and output client_dns_lookup parameter to allow
control of how DNS requests are made
Rabbitmq Integration - 7.1.0
* Added support in Output plugin for sprintf templates in values
provided to message_properties
* Added support for extended metadata including raw payload to events
generated by the Input Plugin
* Fixes an issue with custom port assignment, in which the custom
port was not being applied when more than one host was supplied
* Fixes bug where attempting to read from undeclared exchange
resulted in infinite retry loop
* Fixes bug where failing to establish initial connection resulted in
a pipeline that refused to shut down
Elastic_app_search Output - 1.1.0
* Switched AppSearch client library from Java to Ruby
* Covered with integration tests and dockerized local AppSearch
server instance.
Elasticsearch Output - 10.6.1
* Fixed an issue introduced in 10.6.0 that broke Logstash Core's
monitoring feature when this plugin is run in Logstash 7.7-7.8.
* Added ecs_compatiblity mode, for managing ECS-compatable templates
Testport on 12-STABLE:
https://freebsd-stable.builder.wilbury.net/data/12_STABLE_GENERIC_amd64-default
/2020-08-23_09h37m26s/logs/logstash7-7.9.0.log
Testport on 13-CURRENT:
https://freebsd-current.builder.wilbury.net/data/13_CURRENT_GENERIC_amd64-defau
lt/2020-08-23_09h41m30s/logs/logstash7-7.9.0.log
More information about the freebsd-elastic
mailing list