Elasticsearch - Migrate data from one large index to many time-based indicies

Question

We have one large index on 5 shards on a 5.5 cluster. Each shard has several hundred GB of time-series data. It's not log data for debugging, we're serving production requests doing aggregations on the documents. So we need solid read performance on these aggregations, and enough indexing performance to keep up with the flow of new records.

Most of the aggregation queries probably need to only go over a subset of the data (e.g. the last 30 days of it), so we'd like to migrate the data to an index configuration where we have one index per day, or something similar so that we can increase the number of indexes / shards and scale more horizontally.

How do you migrate your data from one large index to 500 daily indexes? I don't see an API in ES that lets you do this. Can ILM do this during a re-index?

Answer 1

if it's time based data, logs or something else, then treat it the same and use time based indices

ILM is the best longer term approach here, but a reindex into an ILM index/policy won't really work as it works of index ages, not the age of data in those indices

you could do a historic reindex into time based indices, and then point new data into an ILM policy, that'd be the best approach