'Avoid using scroll in ReactiveElasticsearchTemplate or Clearing/closing scroll after use
We have a spring webflux application which is querying Elasticsearch using ReactiveElasticsearchTemplate like this
final inline fun <reified R> getSearchMonoList(
indexName: String,
query: NativeSearchQuery,
metricName: String
): Mono<List<SearchHit<R>>> {
val startTime = getCurrentTime()
recordThroughput(metricName, THROUGHPUT)
return when (indexName == EMPTY_STRING) {
true -> getEsClientTemplate().search(query, R::class.java).collectList()
else -> getEsClientTemplate().search(query, R::class.java, IndexCoordinates.of(indexName)).collectList()
}.doOnError {
recordThroughput(metricName, FAILED_THROUGHPUT)
}.doFinally {
getEsClientTemplate()
recordTime(metricName, startTime)
}
}
Client Configuration is
public ReactiveElasticsearchClient reactiveElasticsearchClient() {
ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo(urls.toArray(String[]::new))
.withWebClientConfigurer(webClient -> {
final ExchangeStrategies exchangeStrategies = ExchangeStrategies.builder()
.codecs(configurer -> configurer.defaultCodecs()
.maxInMemorySize(-1))
.build();
return webClient.mutate().exchangeStrategies(exchangeStrategies).build();
})
.build();
return ReactiveRestClients.create(clientConfiguration);
}
Our Problem
whenever Template queries ES Request body: {"from":0,"size":10000} is requesting ES for 10000 records which was way too much for a term query so we have fixed in query builder by using Pageable, However we do not want to use scroll as this is exhausting Max scroll connections as this API is very heavily used.
NativeSearchQueryBuilder().withQuery(boolQueryBuilders).withPageable(PageRequest.of(0,1))
i am aware of this spring documentation where this was the suggested solution
SearchScrollHits<SampleEntity> scroll = template.searchScrollStart(1000, searchQuery, SampleEntity.class, index);
String scrollId = scroll.getScrollId();
List<SampleEntity> sampleEntities = new ArrayList<>();
while (scroll.hasSearchHits()) {
sampleEntities.addAll(scroll.getSearchHits());
scrollId = scroll.getScrollId();
scroll = template.searchScrollContinue(scrollId, 1000, SampleEntity.class);
}
template.searchScrollClear(scrollId);
However in current production we can not update the libraries because of DMZ restrictions, We are using spring data elasticsearch 4.1.1
How can i disable scroll or clear the scroll after use any help would be appreciable ?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
