Is it possible to define two ILM policies in single filebeat.yml

Question

I'd like to know if it is feasible to configure FileBeat in following way:

I have single rabbitmq with several MQTT topics. I'd like to save messages from topics foo/# in index index-foo and messages from topics bar/# in index index-bar in order to set up separate rolling policies.

I managed to split MQTT messages into separate indexes by setting index property on filebeat.inputs but I can't apply rolling policies then.

Moreover ILM syntax

setup.ilm:
  enabled: true
  rollover_alias: "jkk-logs-filebeat-yml-rollover_alias"
  pattern: "{now/d}-000001"
  overwrite: true
  policy_name: "jkk-policy"
  policy_file: /usr/share/filebeat/ilm.json

seams to allow to set up only one policy.

I'd like to avoid running two separate FileBeat instances.

Is it possible? Can anyone point me to a tutorial or GitHub repository which shows how to do that?

I'm using ES and FB version 7.17.0 (if it does matter).

Answer 1

there's no way to handle multiples like that and you will need to setup the ilm policies directly in Elasticsearch, and just tell Filebeat to send to the alias

Answer 2

As pointed out by Mark, you should do the following,

create two index templates with their index patterns and ILM policies on Elasticsearch. on the Filebeat, disable ILM setup and tell it to send the data to the alias.

if you can't use different alias on the Filebeat , you can use ingest pipeline on both index template to change the target index name based on some values.