Snapshot Backup is not complete deleting form Elastic-Search cluster with SLM policy

Question

I have three nodes ELK-Cluster, I have created an SLM policy to take a Snapshot backup on the daily basis and delete the snapshot backup after three days on the cluster [Issue]: The snapshot backup is deleted from one node only but it is not deleting the snapshot backup from reaming two nodes.

Please find the below steps also. create directory on each node.

 /etc/elasticsearch/ mkdir elkbackup

Give the permission of the directory on each node chown elasticsearch elkbackup

update the elasticserch.yml file on each node for the backup path

path.repo: ["/etc/elasticsearch/elkbackup"]

Restart the each elasticserch service systemctl restart elasticserch

create a snapshot on master or slave node:

curl -u elastic:elastic123 -XPUT 
   -H "Content-Type: application/json" 'http://localhost:9200' 
   -d' {"type":"fs","settings":{"location":"/etc/elasticsearch/elkbackup","compress":"true"}}'

create snapshot life-cycle on master or salve node:

curl -u elastic:elastic123 -XPUT 
   -H "Content-Type: application/json" 'http://localhost:9200/_slm/' 
   -d'{ "schedule": "0 0 2 * * ?", "name": "<elk-backup-{now/d}>", "repository": "elkrepo", "config": {"indices": ["*"]}, "retention": {"expire_after": "3d"} }'

Answer 1

TLDR;

I believe you created 3 filesystem repository, one on each node. You should only have a single one shared by all 3 nodes.

To understand

You are using the shared file system repository. But as the name suggest, all master node should have access to the same filesystem. Not just the same path.

You need to mount a single filesystem on all the master node at the same mountpoint.