'Elasticsearch OutOfMemoryError Java heap space

I am running an 8 cores, 32g RAM elasticsearch node with 5 shards, 400 million (small) documents.

Everything works great until I run an agg search, then shards start failing with: 

java.lang.OutOfMemoryError: Java heap space

I have changed heap size with: export ES_HEAP_SIZE=16g (also ES_MAX_MEM and ES_MIN_MEM to same)

also changed the yml file for elasticsearch:

bootstrap.mlockall: true

and even (recommended by install documents):

sudo sysctl -w vm.max_map_count=262144

Restart service and still no no impact, still java.lang.OutOfMemoryError: Java heap space

Any other suggestions? other than don't run agg queries?

query is: 

https://localhost:9200/my_index_name/_search?search_type=count
{
  "aggs": {
    "distinct_hostname": {
      "cardinality": {
        "field": "hostname"
      }
    }
  }
}
javaout-of-memoryheap-memory


Solution 1:[1]

I think I have discovered the error. I was using 'service' to run elasticsearch and therefore my environment variables got stripped. I had to update the /etc/default/elasticsearch file with the correcct env variables (specifically the ES_HEAP_SIZE=16g).

So far it's running well and app is not erroring.

Solution 2:[2]

The correct way to update Java heap size for Elasticsearch 5 is not EXPORT _JAVA_OPTIONS or EXPORT ES_HEAP_SIZE or using command line parameters. As far as I can tell, all of these are overridden by a configuration file in your Elasticsearch install directory, config/jvm.options. To change these settings you need to edit these lines in that file:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms2g
-Xmx2g

Solution 3:[3]

Although this is an old question, here is how you fix this in newer versions of ELK (Elasticsearch, Kibana, Logstash) (v 5.0+):

Configure your jvm.options file to allocate more memory on the heap.
The options you will need to configure are the -Xms and -Xmx options, which by default are set to 1 Gigabyte of RAM (-Xms1G and -Xmx1G). The recommended size for this is no more than half of your system RAM. For example, on a PC with 32GB RAM, you would set set the following: -Xms16G and -Xmx16G.

This file can be found at /etc/elasticsearch/jvm.options on most ELK installs.

The docs have much more detailed info: https://www.elastic.co/guide/en/elasticsearch/reference/master/advanced-configuration.html#set-jvm-heap-size

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Matt Ball
Solution 2 Craig
Solution 3 Prithvi Boinpally

Related Questions