'Prometheus is failing to send data in TSDB

I have to clusters A and B. In cluster A is Prometheus running and in cluster B is running a service as the long term storage for Prometheus. Since I am trying to make service communication across multiple cluster, I decided to use Linkerd service mesh. Linkerd service mesh for multi-cluster is installed on each cluster. I mirrored the service that runs in cluster B. The added the fqdn to Prometheus remoteWrite as follows:

prometheus:
  prometheusSpec:
    remoteWrite:
    - url: http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local:8428/api/v1/write

AFAIU with Linkerd mirroring the service can communicate across clusters. I am getting

ts=2022-03-21T11:06:00.511Z caller=dedupe.go:112 component=remote level=warn remote_name=32d440 url=http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local:8428/api/v1/write msg="Failed to send batch, retrying" err="Post \"http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local:8428/api/v1/write\": dial tcp: lookup vmsingle-victoria-metrics-single-server.default.svc.cluster.local on 10.0.0.10:53: no such host"

It seems to the communication is not working as I expected. Where do I do wrong?

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source

'Prometheus is failing to send data in TSDB

Sources

Related Questions