Kubernetes coredns is not reachable from the pod

Question

I have a pod deployed named 'sample_pod' in rancher cluster having a container named 'sample_container'. The sample pod has a service named 'test'. Inside the sample_container, if I try to resolve the cluster domain names using 'host' or 'dig' or 'nslookup' command, I am always getting connection refused; no servers could be reached.

I have coredns pods running inside my cluster

user@abc$ kubectl get pods -n kube-system
NAME                                       READY   STATUS      RESTARTS   AGE
calico-kube-controllers-7fbff695b4-f7vxc   1/1     Running     0          21h
canal-928m6                                2/2     Running     0          21h
canal-d7vjr                                2/2     Running     0          20h
coredns-6f85d5fb88-9txmx                   1/1     Running     0          21h
coredns-autoscaler-79599b9dc6-ndgfj        1/1     Running     0          21h
kube-multus-ds-769n6                       1/1     Running     0          20h
metrics-server-8449844bf-jz66w             1/1     Running     0          21h
rke-coredns-addon-deploy-job-dlvlh         0/1     Completed   0          21h
rke-ingress-controller-deploy-job-jcj6w    0/1     Completed   0          21h
rke-metrics-addon-deploy-job-wnhbq         0/1     Completed   0          21h
rke-network-plugin-deploy-job-wzqfb        0/1     Completed   0          21h
whereabouts-p6vcc                          1/1     Running     0          20h

I am not touching the default Corefile of coredns

Corefile:
.:53 {
  log
  errors
  health {
    lameduck 5s
  }
  ready
  kubernetes cluster.local in-addr.arpa ip6.arpa {
    pods insecure
    fallthrough in-addr.arpa ip6.arpa
  }
  prometheus :9153
  forward . "/etc/resolv.conf"
  cache 30
  loop
  reload
  loadbalance
}

/etc/hosts file of sample_container:

[root@sample_container]# cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.42.1.18      sample_pod      

# Entries added by HostAliases.
127.0.0.1       localhost
10.94.66.8      netboot.com 

/etc/resolv.conf of sample_container:

[root@sample_container]# cat /etc/resolv.conf
nameserver 10.43.0.10
search default.svc.cluster.local svc.cluster.local cluster.local openstacklocal
options ndots:5

Host or dig command I used to resolve following domains and got the error:

[root@sample_container]# ping 10.43.0.10
PING 10.43.0.10 (10.43.0.10) 56(84) bytes of data.
^C
--- 10.43.0.10 ping statistics ---
99 packets transmitted, 0 received, 100% packet loss, time 98003ms

[root@sample_container]# host kube-dns.kube-system
;; connection timed out; no servers could be reached

[root@sample_container]#  host localhost
;; connection timed out; no servers could be reached

I tried to resolve test service in the default namespace (where sample_container, sample_pod resides in same namespace) 
[root@sample_container]# host test
;; connection timed out; no servers could be reached


dig or nslookup command also returns same
[root@sample_container]# nslookup localhost
;; connection timed out; no servers could be reached

[root@sample_container]# dig localhost

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.8 <<>> localhost
;; global options: +cmd
;; connection timed out; no servers could be reached

Additional information on pod ip and service ip:

root@user$ kubectl get all -o wide
NAME           READY   STATUS    RESTARTS   AGE    IP           NODE                      NOMINATED NODE   READINESS GATES
pod/sample_pod       1/1     Running   0          177m   10.42.1.18   dsc-worker-node    <none>           <none>

NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                           AGE    SELECTOR
service/test           ClusterIP   10.43.19.85   <none>        80/TCP,443/TCP                 177m   role=test

Note: I deployed this pod such a way that some containers will access the baremetal machine to serve its purpose. And I need to achieve forwarding certain domain names to that baremetal server which will reply for that dns query. Also I am aware of forward plugin which does this job. But without touching the Corefile, I am unable to reach coredns for cluster domain names itself.

Could someone help me to solve this issue? It would be really helpful for me. Thanks in advance!!!

Answer 1

I solved this issue after changing the route. By default, the dns queries are sent to kubernetes nameserver via private interface instead of sending via default gateway (public interface). After changing the route to make dns queries to be sent via default gateway, it was solved.