Implements popular keyword in ElasticSearch

Question

I'm using ElasticSearch on AWS EC2. And i want to implement today's popular keyword function in ES. there is 3 indexes(place, genre, name), and i want see today's popular keyword in name index only.

I tried to use ES slowlog and logstash. but slowlog save logs every shard's log. (ex)number of shards : 5 then 5 query log saved.

Is there any good and easy way to implement popular keyword in ES?

Answer 1

As far as I know, this is not supported by Elasticsearch and you need to build your own custom solution.

Design you mentioned using the slowlog is not good as you mentioned its on per shard basis, even if you do some more computing and able to merge and relate them to a single search at index level, it would not be good, as

1. you have to change the slow log configuration and for every index there needs to be a different threshold, you can change it to 0ms, to make sure you get all the search queries in slow logs, but that would take a huge disk space and would not be good for Elasticsearch performance.
2. You have to do some parsing of slow log in your application and if you do it runtime it would be very costly.

I think you can maintain a distributed cache in your application where you store the top searched keyword like the leaderboard of a multi-player gaming app, which is changing very frequently but in your case, you don't even have to update this cache very frequently. I would not go into much implementation details, but simple Hashmap of search term as key and count as value would solve the issue.

Hope this helps. let me know if you have questions.