Elasticsearch snapshot restore from S3 failed with RepositoryMissingException

Question

I was able to create the repository successfully, and list the snapshots. That showed the repository couldn't be missing.

Yet the restore request failed with RepositoryMissingException, with the following details.

shard has failed to be restored from the snapshot [lb-es-snapshots:snapshot-
2022-01-09/gnA_ObsiRmOA-ydXfZWfbA] because of [failed shard on node [RsQQ6-
L6R_6qTIJigizMXQ]: failed recovery, failure RecoveryFailedException[[api][0]: Recovery 
failed on {my-release-elasticsearch-data-0}{RsQQ6-L6R_6qTIJigizMXQ}
{w3E20XKZTHyAvpI7XEogjQ}{my-release-elasticsearch-data-0.my-release-elasticsearch-data-
hl.default.svc.cluster.local}{10.244.1.24:9300}{d}{xpack.installed=true, 
transform.node=false}]; nested: RepositoryMissingException[[lb-es-snapshots] missing]; ] 
- manually close or delete the index [api] in order to retry to restore the snapshot 
again or use the reroute API to force the allocation of an empty primary shard

Is there a way to make sense of the error? the logs on the nodes show the same exception.

Answer 1

answering my own question

before registering the repository, you need to manually add the S3 access and secret key to the keystore on BOTH (this is not mentioned in ES documentation) the first master node and the first data node, and reload the settings.