How to rotate ELK logs?

Question

I have indexes around 250 GB all-together in 3 host i.e. 750 GB data in ELK cluster.

So how can I rotate ELK logs to keep three months data in my ELK cluster and older logs should be pushed some other place.

Answer 1

Answer by dexter_ is correct, but as the answer is old, a better answer would be:

version 7.x of elastic stack provides a index life cycle management policies, which can be easily managed with kibana GUI and is native to elk stack. PS, you still have to manage the indices like "indexname-%{+YYYY.MM}" as suggested dexter_

elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html

Answer 2

You could create your index using "indexname-%{+YYYY.MM}" naming format. This will create a distinct index every month.

You could then filter this index, based on timestamp, using a plugin like curator. The curator could help you set up a CRON job to purge those older indexes or back them up on some s3 repository.

Reference - Backup or Restore using curator

Moreover, you could even restore these backup indexes whenever needed directly from s3 repo for historical analysis.

Answer 3

It took me a while to figure out exact syntax and rules, so I'll post the final policy I used to remove old indexes (it's based on the example from https://aws.amazon.com/blogs/big-data/automating-index-state-management-for-amazon-opensearch-service-successor-to-amazon-elasticsearch-service/):

{
    "policy": {
        "description": "Removes old indexes",
        "default_state": "active",
        "states": [
            {
                "name": "active",
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "14d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": {
            "index_patterns": [
                "mylogs-*"
            ]
        }
    }
}

It will automatically apply the policy for any new mylogs-* indexes, but you'll need to apply it manually for existing ones (under "Index Management" -> "Indices").