Connection between pods in k8s (etcd cluster)

Question

I am trying to set up a cluster of etcd pods in k8s.

I am doing this.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-etcd-storage-class
  namespace: etcd
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: etcd
  namespace: etcd
spec:
  serviceName: etcd
  replicas: 2
  selector:
    matchLabels:
      app: etcd
  template:
    metadata:
      labels:
        app: etcd
    spec:
      containers:
      - name: etcd
        image: gcr.io/etcd-development/etcd:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 2379
          name: client
        - containerPort: 2380
          name: peer
        volumeMounts:
        - name: etcd-claim-template
          mountPath: /var/run/etcd
        command:
          - /bin/sh
          - -c
          - |
            echo ${HOSTNAME}
            PEERS="etcd-0=http://etcd-0.etcd:2380,etcd-1=http://etcd-1.etcd:2380,etcd-2=http://etcd-2.etcd:2380"
            exec etcd --name ${HOSTNAME} \
              --listen-peer-urls http://0.0.0.0:2380 \
              --listen-client-urls http://0.0.0.0:2379 \
              --advertise-client-urls http://${HOSTNAME}.etcd:2379 \
              --initial-advertise-peer-urls http://${HOSTNAME}:2380 \
              --initial-cluster-token etcd-cluster-1 \
              --initial-cluster ${PEERS} \
              --initial-cluster-state new \
              --data-dir /var/run/etcd/default.etcd
  volumeClaimTemplates:
  - metadata:
      name: etcd-claim-template
    spec:
      storageClassName: local-etcd-storage-class
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

---
apiVersion: v1
kind: Service
metadata:
  name: etcd-svc
  namespace: etcd
spec:
  selector:
    app: etcd
  clusterIP: None
  ports:
  - name: etcd-client-port
    port: 2379
    protocol: TCP
  - name: etcd-peer-port
    port: 2380
    protocol: TCP

The problem I am facing is that etcd-0.etcd does not see etcd-1.etcd

2022-05-04 12:23:21.803307 W | rafthttp: health check for peer 5ba2a9fe8d840508 could not connect: dial tcp: lookup etcd-2 on 10.245.0.10:53: no such host (prober "ROUND_TRIPPER_SNAPSHOT")
2022-05-04 12:23:21.803509 W | rafthttp: health check for peer eba8308136b9bf9e could not connect: dial tcp: lookup etcd-1 on 10.245.0.10:53: no such host (prober "ROUND_TRIPPER_SNAPSHOT")
2022-05-04 12:23:21.803737 W | rafthttp: health check for peer eba8308136b9bf9e could not connect: dial tcp: lookup etcd-1 on 10.245.0.10:53: no such host (prober "ROUND_TRIPPER_RAFT_MESSAGE")
2022-05-04 12:23:21.804521 W | rafthttp: health check for peer 5ba2a9fe8d840508 could not connect: dial tcp: lookup etcd-2 on 10.245.0.10:53: no such host (prober "ROUND_TRIPPER_RAFT_MESSAGE")

Is there a way to make two pods communicate? Do I need separate services for each pod replica?

I am stuck with this though I thought it will work as it is.

I tried to ping etcd-1 from etcd-0 but is does not see the domain.

etcd-0 terminal

ping etcd-1
ping: bad address 'etcd-1'

ping etcd-1.etcd
ping: bad address 'etcd-1.etcd'

ping etcd-1.etcd.etcd.svc.cluster.local
ping: bad address 'etcd-1.etcd.etcd.svc.cluster.local'

ping etcd-1.etcd.svc.cluster.local
ping: bad address 'etcd-1.etcd.svc.cluster.local'

DO I need to do something to see other pods from the same stafullset