Elasticsearch with fluent doesn't add logs to the new rollover index

Question

I'm using Elastic Search with Fluent and I set up a ILM for the indices. I have the following policy:

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "2d",
            "max_size": "50mb",
            "max_docs": 50
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "2m",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

For the first index it works fine, but my problem is when the new rollover index is created, it still adds the logs to the initial index. Any ideas what I got wrong? I would like the logs to be added to the last index created.

fluent.conf

<match *.**>
  @type copy
  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format false
    index_name fluentd-log-000001
    type_name access_log
    tag_key @log_name
    flush_interval 5s
  </store>

  <store>
    @type stdout
  </store>
</match>

template

PUT _template/fluentd-template
{
  "index_patterns": [
    "fluentd*"
  ],
  "settings": {
    "index.lifecycle.name": "fluentd-policy",
    "index.lifecycle.rollover_alias": "fluentd"
  },
  "mappings": {
    "_source": {},
    "_meta": {},
    "properties": {}
  }
}

Answer 1

Take a look at this specific part of your fluentd.conf configuration:

  index_name fluentd-log-000001

You're explicit specifying which index should receive the logs: fluentd-log-000001. That's the reason why fluentd is sending always to the same index (take a look at the official documentation).

One nice approach that you can do is: just let fluentd create the index for you and remove the rollover_alias from your template (it's not going to be necessary anymore). Then, add the following parameters to your fluentd.conf file:

  ilm_policy_id <your-ilm-policy-name>
  template_name <your-index-template-name>
  logstash_format true
  logstash_prefix fluentd-log
  logstash_dateformat %Y.%m.%d

When logstash_format is true, it supersedes the index_name option. With the configuration above, your new indexes will be created with a date suffix just like the following:

fluentd-log-2022.02.25

All new indexes will follow the ILM policy and the index template specified.