Elasticsearch Doucment disappears for a few seconds

Question

After performing a _bulk operation to insert 50 documents, I run a search. I noticed that sometimes the search doesn't return all 50 documents. Sometimes somewhere between 1 and 3 documents will be missing. I have a polling mechanism that finds all 50, but it seems sometimes even after successful polling, the follow-up search query is short. But, given enough time (usually just an extra second or two), all 50 do become available.

The documentation says that if you want to run a search immediately after an insert, you should use the ?refresh=wait_for parameter, but my team really wants to avoid that as it would require a lot of re-testing and refactoring.

As an alternative, I tried running a simple polling query to make sure all 50 documents got into the index, and then I notify the service that runs the complicated query that it can run its query now.

However, even with the polling - sometimes the polling will be successful and find 50 documents, but the follow-up query will still have a few documents missing. I can even see this when I constantly run my query in ES Dev Tools after I do the bulk insert. I see the documents populate gradually - but right after I see 50 documents, the next query will be 48, and then right after it will be 50 again.

So, all 50 get in at first (which is why my polling is successful), but it seems there is a short period where some documents will disappear from search, but then they reappear.

Is this normal behavior? Is there anything I can do besides ?refresh=wait_for?

Answer 1

You can use on _bulk call

• ?refresh=wait_for - Wait for the changes made by the request to be made visible by a refresh before replying.
• ?refresh=true - Refresh the relevant primary and replica shards (not the whole index) immediately after the operation occurs, so that the updated document appears in search results immediately.

Or, if not possible to amend the _bulk call, you can explicitly call POST <target>/_refresh to refresh the index, see https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html