Elasticsearch: Job for elasticsearch.service failed

Question

I am currently trying to setup Elasticsearch for a project. I have installed Elasticsearch 7.4.1 and I have also installed Java, that is openjdk 11.0.4.

But when I try to start Elasticsearch using the command

sudo systemctl start elasticsearch

I get the error below

Job for elasticsearch.service failed because the control process exited with error code.

See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

And when I try to run the command

systemctl status elasticsearch.service

I get the error message

● elasticsearch.service - Elasticsearch

Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vend

Active: failed (Result: exit-code) since Fri 2019-11-01 06:09:54 UTC; 12s ago

Docs: http://www.elastic.co

Process: 5960 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DI

Main PID: 5960 (code=exited, status=1/FAILURE)

I have removed/purged Elasticsearch from my machine and re-installed several times, but it doesn't seem to fix the issue.

I have tried to modify the default network.host and host.port settings in /etc/default/elasticsearch to network.host: 0.0.0.0 and http.port: 9200 to fix the issue, but no luck yet.

I need some assistance. Thanks in advance.

Answer 1

Here's how I solved

Firstly, Open /etc/elasticsearch/elasticsearch.yml in your nano editor using the command below:

sudo nano /etc/elasticsearch/elasticsearch.yml

Your network settings should be:

# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 127.0.0.1
#
# Set a custom port for HTTP:
#
http.port: 9200

In order for Elasticsearch to allow connections from localhost, and to also listen on port 9200.

Next, run the code below to determine the cause of the error:

journalctl -xe

Error 1

There is insufficient memory for the Java Runtime Environment to continue

Solution

As a JVM application, the Elasticsearch main server process only utilizes memory devoted to the JVM. The required memory may depend on the JVM used (32- or 64-bit). The memory used by JVM usually consists of:

• heap space (configured via -Xms and -Xmx)
• metaspace (limited by the amount of available native memory)
• internal JVM (usually tens of Mb)
• OS-dependent memory features like memory-mapped files.

Elasticsearch mostly depends on the heap memory, and this setting manually by passing the -Xms and -Xmx(heap space) option to the JVM running the Elasticsearch server.

Solution

Open /etc/elasticsearch/jvm.options in your nano editor using the command below:

sudo nano /etc/elasticsearch/jvm.options

First, un-comment the value of Xmx and Xms

Next, modify the value of -Xms and -Xmx to no more than 50% of your physical RAM. The value for these settings depends on the amount of RAM available on your server and Elasticsearch requires memory for purposes other than the JVM heap and it is important to leave space for this.

Minimum requirements: If your physical RAM is <= 1 GB

Then, your settings should be:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms128m
-Xmx128m

OR

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms256m
-Xmx256m

Medium requirements: If your physical RAM is >= 2 GB but <= 4 GB

Then, your settings should be:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms512m
-Xmx512m

OR

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms750m
-Xmx750m

Large requirements: If your physical RAM is >= 4 GB but <= 8 GB

Then, your settings should be:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms1024m
-Xmx1024m

OR

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms2048m
-Xmx2048m

Note: If your physical RAM is >= 8 GB you can decide how much heap space you want to allocate to Elasticsearch. You can allocate -Xms2048m and -Xmx2048m OR -Xms4g and -Xmx4g or even higher for better performance based on your available resources.

Error 2

Initial heap size not equal to the maximum heap size

Solution

Ensure the value of -Xms and Xmx are equal. That is, say, you are using the minimum requirements since your physical RAM is <= 1 GB, instead of this:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms128m
-Xmx256m

it should be this:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms128m
-Xmx128m

OR this:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms256m
-Xmx256m

Error 3

the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

Solution

Open /etc/elasticsearch/elasticsearch.yml in your nano editor using the command below:

sudo nano /etc/elasticsearch/elasticsearch.yml

Your discovery settings should be:

# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: []

Once all the errors are fixed run the command below to start and confirm the status of Elasticsearch:

sudo systemctl start elasticsearch
sudo systemctl status elasticsearch

That's all.

I hope this helps

Answer 2

Had the same problem with a small virtual machine. The above configurations were already set. The only thing that helped was to increase the start timeout. The standard systemd timeout was just not enough.

As a precaution, I set the timeout to 5 minutes as follows.

sudo nano /usr/lib/systemd/system/elasticsearch.service

Added under [Service] section in elasticsearch.service file.

TimeoutStartSec=300

Activate change to service.

sudo /bin/systemctl enable elasticsearch.service

Start service again.

service elasticsearch start

Answer 3

A basic solution to this problem is to just uninstall Elasticsearch and Kibana and again re-install them and your problem will be solved.

For uninstalling Elasticsearch:

sudo apt-get remove --purge elasticsearch

The message was:

dpkg: warning: while removing elasticsearch, directory '/var/lib/elasticsearch' not empty so not removed
dpkg: warning: while removing elasticsearch, directory '/etc/elasticsearch' not empty so not removed

Removed those directories as well:

sudo rm -rf /etc/elasticsearch
sudo rm -rf /var/lib/elasticsearch

Then install it again:

sudo apt-get install elasticsearch=7.10.1
sudo systemctl start elasticsearch
curl http://localhost:9200/

For uninstalling Kibana:

sudo apt-get remove --purge kibana

Removed those directories as well:

sudo rm -rf /etc/kibana
sudo rm -rf /var/lib/kibana

Then install it again:

sudo apt-get install kibana=7.10.1
sudo systemctl start kibana

For opening Kibana on browser:

http://localhost:5601

Answer 4

If you installed using package management, check if the owner of /etc/elasticsearch directory is elasticsearch.

sudo chown -R elasticsearch:elasticsearch /etc/elasticsearch/

Answer 5

First verify that this is the same problem with command:

journalctl -xe

If you see error like this java.lang.NoClassDefFoundError: Could not initialize class then do this:

My solution I got from here https://github.com/elastic/elasticsearch/issues/57018

sudo nano /etc/sysconfig/elasticsearch

Add this at the end or beggining of the file

# Elasticsearch temp directory
ES_TMPDIR=/var/log/elasticsearch

Answer 6

Try a system restart or just logging-out

I had the same issue as OP on a fresh install of ES. Before going down a rabbit hole of logs and Google searches, I simply tried logging-out of my OS (Ubuntu 20.04) and logged back in. Opened a fresh terminal and elasticsearch was able to start successfully.

For reference, I used:

sudo service elasticsearch restart

and checked it with:

sudo service elasticsearch status

Answer 7

In my case, java was missing from my server When I have reconfigured my new server I did not check java.

After install java, it starts working.

It may help someone

Please first check java is pre-installed or not... because it is a pre-requirement of elasticsearch.

# systemctl status elasticsearch.service
# which java
# locate java
# java --version
# sudo apt install openjdk-11-jre-headless
# java --version
# sudo systemctl stop elasticsearch
# sudo systemctl start elasticsearch

Thank you.

Answer 8

Steps to install elasticsearch 7.15.2 Follow this digital ocean article

If you see this error

Job for elasticsearch.service failed because the control process exited with error code.

See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

Open sudo nano /etc/elasticsearch/elasticsearch.yml

Un comment these

network.host: 127.0.0.1
http.port: 9200

Open sudo nano /etc/elasticsearch/jvm.options

Un comment these

-Xms4g 
-Xmx4g

Open sudo nano /etc/elasticsearch/elasticsearch.yml

Update this

discovery.seed_hosts: []

At last run these

sudo systemctl start elasticsearch
sudo systemctl status elasticsearch

To check if its working or not run this command

curl -X GET 'http://localhost:9200'

Answer 9

I am using ubuntu 20.04 and in my case, the issue was with the installation part. I followed automatic installation steps in official documentation. After searching for a while, I tried the manual approach described in the same documentation, work like a magic for me.

Answer 10

For some people this might be the case as it was for me so this might help someone.I am noob in writing such things so bear with me.

So got this error

Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: org.elasticsearch.cli.UserException: unable to create temporary keystore a>

Likely root cause: java.nio.file.AccessDeniedException: /etc/elasticsearch/elasticsearch.keystore.tmp

elasticsearch.service: Failed with result 'exit-code'.

elasticsearch.service: Main process exited, code=exited, status=1/FAILURE

for ubuntu 20.04, workaround for this was to run these two commands:

sudo chmod g+w /etc/elasticsearch

Above command changes file permissions (allowing) for creating keystroke manually.And below command create that manually.

sudo -u elasticsearch -s /usr/share/elasticsearch/bin/elasticsearch-keystore create

Answer 11

I also faced the same problem

I checked the elasticsearch service status

sudo systemctl start elasticsearch

Next, run the code below to determine the cause of the error:

journalctl -xe 

I see a lots of line about Performance analyzer

Dec 20 21:07:37 my performance-analyzer-agent-cli[13112]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Dec 20 21:07:37 my performance-analyzer-agent-cli[13112]: Dec 20, 2019 9:07:37 PM org.jooq.tools.JooqLogger info
Dec 20 21:07:37 my performance-analyzer-agent-cli[13112]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Dec 20 21:07:42 my performance-analyzer-agent-cli[13112]: Dec 20, 2019 9:07:42 PM org.jooq.tools.JooqLogger info
Dec 20 21:07:42 my performance-analyzer-agent-cli[13112]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Dec 20 21:07:42 my performance-analyzer-agent-cli[13112]: Dec 20, 2019 9:07:42 PM org.jooq.tools.JooqLogger info
Dec 20 21:07:42 my performance-analyzer-agent-cli[13112]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Dec 20 21:07:47 my performance-analyzer-agent-cli[13112]: Dec 20, 2019 9:07:47 PM org.jooq.tools.JooqLogger info
Dec 20 21:07:47 my performance-analyzer-agent-cli[13112]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Dec 20 21:07:47 my performance-analyzer-agent-cli[13112]: Dec 20, 2019 9:07:47 PM org.jooq.tools.JooqLogger info
Dec 20 21:07:47 my performance-analyzer-agent-cli[13112]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Dec 20 21:07:52 my performance-analyzer-agent-cli[13112]: Dec 20, 2019 9:07:52 PM org.jooq.tools.JooqLogger info
Dec 20 21:07:52 my performance-analyzer-agent-cli[13112]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Dec 20 21:07:52 my performance-analyzer-agent-cli[13112]: Dec 20, 2019 9:07:52 PM org.jooq.tools.JooqLogger info
Dec 20 21:07:52 my performance-analyzer-agent-cli[13112]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Dec 20 21:07:57 my performance-analyzer-agent-cli[13112]: Dec 20, 2019 9:07:57 PM org.jooq.tools.JooqLogger info
Dec 20 21:07:57 my performance-analyzer-agent-cli[13112]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Dec 20 21:07:57 my performance-analyzer-agent-cli[13112]: Dec 20, 2019 9:07:57 PM org.jooq.tools.JooqLogger info
Dec 20 21:07:57 my performance-analyzer-agent-cli[13112]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Dec 20 21:08:02 my performance-analyzer-agent-cli[13112]: Dec 20, 2019 9:08:02 PM org.jooq.tools.JooqLogger info
Dec 20 21:08:02 my performance-analyzer-agent-cli[13112]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Dec 20 21:08:02 my performance-analyzer-agent-cli[13112]: Dec 20, 2019 9:08:02 PM org.jooq.tools.JooqLogger info
Dec 20 21:08:02 my performance-analyzer-agent-cli[13112]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Dec 20 21:08:07 my performance-analyzer-agent-cli[13112]: Dec 20, 2019 9:08:07 PM org.jooq.tools.JooqLogger info
Dec 20 21:08:07 my performance-analyzer-agent-cli[13112]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse
Dec 20 21:08:07 my performance-analyzer-agent-cli[13112]: Dec 20, 2019 9:08:07 PM org.jooq.tools.JooqLogger info
Dec 20 21:08:07 my performance-analyzer-agent-cli[13112]: INFO: Single batch             : No bind variables have been provided with a single statement batch execution. This may be due to accidental API misuse

This makes problem to analyze the issue So First I tried to stop it so I found a link

in /usr/lib/systemd/system/opendistro-performance-analyzer.service

under [Service] add StandardOutput=null

after that reload systemd via ‘/bin/systemctl daemon-reload’ for it to take affect

For more Detail follow the below link https://discuss.opendistrocommunity.dev/t/performance-analyzer-agent-cli-spamming-syslog-in-od-1-3-0/2040/4

Now the picture was clear easily I found the issue that There were duplicate properties in the elasticsearch.yml file that I forgot to comment on. I commented out the duplicate property and restart the elasticsearch service.

sudo systemctl start elasticsearch
sudo systemctl status elasticsearch

That's all.

I hope It will help.