🚀0.5
Released on
Features and Enhancements:
Cache library - This feature is introduced because parsing (and validating and checking privileges) is expensive in terms of CPU cycles.
New Index optimizer node added to optimize/compact indexing in parallel. Now Indexer runs in two modes default mode or “optimizer” mode.
Indexing structure better organized
bucket/folder/stage (For staging files).
bucket/folder/final/YYYYMMDD (Compacted files from optimizer)
Search dynamic caching added for data files and index files.
Two search modes were introduced: “dynamic” where search reads live data from both indexers and S3. Whereas “static” mode will enable loading historical S3-only logs within a given date range.
Bug fixes:
All the Simple Storage Service (s3) calls based issues has been fixed.
To simplify indexer configuration: S3 schema.json loading for indexer is completely removed.
Indexer should be started on a fresh new bucket as the old structure/files are not compatible with the new indexer release.
Config changes:
Configuration Changes Indexer:
ops.server.mode - set to "optimizer" only for the optimizer pod run
data.optimize.schedule.minutes - Frequency of the optimizer run
out.kafka.optimize.topic - The kafka topic where optimize will connect with search
data.process.config.path is removed - Schema.json is removed
schema.tenantid.keys
schema.appid.keys
schema.message.key (defaults: "message")
schema.timestamp.key
schema.timestamp.formats (defaults: "%Y-%m-%dT%H:%M:%S" )
Configuration Changes Search:
ops.index.cache.path - search local disk file path where index caching is done
ops.data.cache.path - search local disk file path where data caching is done
search.mode - “dynamic” or “static”
search.mode.dynamic.kafka.stage.topic
search.mode.dynamic.kafka.final.topic
search.mode.daystoinclude
search.mode.static.fromdate (example: 20201107)
search.mode.static.todate (example: 20201107)
Configuration Changes Gateway:
clusters - Search cluster node
Last updated