Cannot start prometheus by using systemd

Question

OS level: CentOS Linux release 7.4.1708
Prometheus level: 2.4.2
prometheus.service:

[Unit]
Description=Prometheus

[Service]
User=prometheus
ExecStart=/usr/local/prometheus/prometheus

[Install]
WantedBy=default.target

When I use systemctl start prometheus to start the prometheus service, it always exit the main process by itself. And the systemctl's log shows like this:

● prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2018-09-25 10:43:56 CST; 6s ago
  Process: 5174 ExecStart=/usr/local/prometheus/prometheus (code=exited, status=1/FAILURE)
 Main PID: 5174 (code=exited, status=1/FAILURE)

Sep 25 10:43:56 devtestserver systemd[1]: Started Prometheus.
Sep 25 10:43:56 devtestserver systemd[1]: Starting Prometheus...
Sep 25 10:43:56 devtestserver prometheus[5174]: level=info ts=2018-09-25T02:43:56.736457704Z caller=main.go:238 msg="Starting Prometheus" version="(version=2.4.2, branch=HE...13b1190a0)"
Sep 25 10:43:56 devtestserver systemd[1]: prometheus.service: main process exited, code=exited, status=1/FAILURE
Sep 25 10:43:56 devtestserver systemd[1]: Unit prometheus.service entered failed state.
Sep 25 10:43:56 devtestserver systemd[1]: prometheus.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

I have no ideas with this problem. I use the same config for the node_exporter, but node_exporter can start as normal. Please help. Thanks a lot.

Answer 1

You have not added configuration file i.e. prometheus.yml

Considering Service part of your prometheus.service file,

ExecStart=/usr/local/prometheus/prometheus \
--config.file /prometheus-2.26.0.linux-amd64/prometheus.yml

here, my .yml file is in /prometheus-2.26.0.linux-amd64/ location. your might be different. Befre running check your both paths i.e. your executable file is on the path given in "ExecStart" and yml file is in --config.file

then reload your system by

systemctl daemon-reload
systemctl start prometheus
systemctl enable prometheus

then check the status using,

systemctl status prometheus

It should be active(running).

This should solve your problem. Let me know if it helped : )

Answer 2

There is an extra "i" at the end of WantedBy=default.target.

To get more details about services failing to start, try sudo journalctl -ex

My guess is it's either the extra "i" or Prometheus might not be able to parse your scrape rules or alerts files. It comes with "promtool" to check your configuration files and is installed in the same directory as prometheus. Your first step should be to try "promtool check config /path/to/prometheus.yml"

Answer 3

I encountered the same issue with Ubuntu 16.04. Turned out to be a permissions issue. You should check that you user owns the directories in which you installed the binaries and the files inside these directories.

Answer 4

Where is the config file located? systemd is executed by / by default. prometheus reads the setting of ./prometheus.yml by default. Perhaps you need to add the following config option to the unit file of systemd.

[Unit]
Description=Prometheus

[Service]
User=prometheus
ExecStart=/usr/local/prometheus/prometheus --config.file /path/to/your/config

[Install]
WantedBy=default.target

Answer 5

This problem is caused because the data storage directory does not have permission. The default Prometheus data directory is /data. chown -R prometheus:prometheus /data

Answer 6

copy and paste this in your command line:

sudo tee /etc/systemd/system/prometheus.service<<EOF
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP \$MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.external-url=

SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target
EOF

Answer 7

I encountered similar issue in redhat/Centos.I solved it by temporarily running "sudo setenforce 0". You can also edit the /etc/selinux/config file and set the SELINUX to disabled