'ElasticSearch - Getting New Records

I have a Logstash instance that is using the Elasticsearch plugin as an input. This input plugin has a schedule of * * * * *. When it runs, I want to get the records input into my Elasticsearch instance since the last time the scheduler ran? It seems like I need a way to tell the plugin to give me the documents starting when logstash starts. Then, go from there. However, at this time, I'm using the "@timestamp" field in my query and it's getting all of the docs in the Elasticsearch index since the time my Elasticsearch index was started.

How do I get documents in an Elasticsearch index beginning when Logstash starts, and then get the new documents since the last time the schedule ran?

I've been beating my head on this one without any luck. Thank you so much in advance for your help!

logstash


Solution 1:[1]

From what I read, it may not be possible with much precision. The best I see is to run the query every minute and have the query only look back one minute. Obviously there are lots of edge cases which will result in missing and duplicate data.

The JDBC plugin supports last_run_metadata_path to store the last time, and can make this available to the query as :sql_last_start. I don't see that anyone has ever requested this feature for the ES input plugin.

Solution 2:[2]

I have been also going to get same things as mentioned above,but i've have got result about sql but no yet for the elastic search

{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "last_run_metadata_path"
      }
    }
  }
}

this should be work

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Alain Collins
Solution 2 1sina1

Related Questions