'CoreDNS returning incorrect IP (mostly)
Having difficulties tracking down this issue. Have had my cluster running untouched for about 6 months with no issues. It reboots on the first of the month, every month. As of May the 1st after it rebooted there have been issues with connectivity between my services. Mainly nginx reverse proxies, but it's happening between all services (Authelia to LDAP and so on).
After much investigation it seems that pods are resolving the incorrect IP addresses for services. When querying CoreDNS directly from inside an nginx pod, I can clearly see the IP addresses of my services swapping. For example:
/ # cat /etc/resolv.conf
nameserver 10.96.0.10
search cert-manager.svc.cluster.local svc.cluster.local cluster.local mydomain.xyz
options ndots:5
/ # nslookup motioneye.home.svc.cluster.local 10.96.0.10
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: motioneye.home.svc.cluster.local
Address 1: 10.105.38.36 motioneye.home.svc.cluster.local
/ # nslookup motioneye.home.svc.cluster.local 10.96.0.10
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: motioneye.home.svc.cluster.local
Address 1: 10.7.9.10 quickstart-ingress-nginx-controller.cert-manager.svc.cluster.local
Logs from CoreDNS:
[INFO] 10.244.2.187:57366 - 13803 "AAAA IN motioneye.home.svc.cluster.local. udp 50 false 512" NOERROR qr,aa,rd 143 0.000276052s
[INFO] 10.244.2.187:57366 - 13462 "A IN motioneye.home.svc.cluster.local. udp 50 false 512" NOERROR qr,aa,rd 98 0.000533974s
[INFO] 10.244.2.187:59445 - 12082 "PTR IN 36.38.105.10.in-addr.arpa. udp 43 false 512" NOERROR qr,aa,rd 114 0.00028283s
[INFO] 10.244.2.187:34510 - 42594 "AAAA IN motioneye.home.svc.cluster.local.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 157 0.000262126s
[INFO] 10.244.2.187:34510 - 42137 "A IN motioneye.home.svc.cluster.local.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 157 0.000489475s
[INFO] 10.244.2.187:48887 - 48126 "AAAA IN motioneye.home.svc.cluster.local.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 157 0.000580881s
[INFO] 10.244.2.187:48887 - 47758 "A IN motioneye.home.svc.cluster.local.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 157 0.000857914s
[INFO] 10.244.2.187:57896 - 20940 "A IN motioneye.home.svc.cluster.local.mydomain.xyz. udp 61 false 512" NOERROR qr,aa,rd,ra 120 0.009537425s
[INFO] 10.244.2.187:57896 - 21303 "AAAA IN motioneye.home.svc.cluster.local.mydomain.xyz. udp 61 false 512" NXDOMAIN qr,rd,ra 143 0.030400843s
[INFO] 10.244.2.187:43096 - 9412 "PTR IN 10.9.7.10.in-addr.arpa. udp 40 false 512" NOERROR qr,aa,rd 142 0.000471438s
[INFO] 10.244.2.187:52860 - 25652 "AAAA IN motioneye.home.svc.cluster.local.mydomain.xyz. udp 61 false 512" NOERROR qr,aa,rd,ra 61 0.000276867s
[INFO] 10.244.2.187:52860 - 24926 "A IN motioneye.home.svc.cluster.local.mydomain.xyz. udp 61 false 512" NXDOMAIN qr,aa,rd,ra 143 0.000415013s
[INFO] 10.244.2.187:48476 - 38937 "A IN motioneye.home.svc.cluster.local. udp 50 false 512" NOERROR qr,aa,rd 98 0.000482234s
[INFO] 10.244.2.187:48476 - 39260 "AAAA IN motioneye.home.svc.cluster.local. udp 50 false 512" NOERROR qr,aa,rd 143 0.00066462s
[INFO] 10.244.2.187:53694 - 43277 "PTR IN 36.38.105.10.in-addr.arpa. udp 43 false 512" NOERROR qr,aa,rd 114 0.000234164s
[INFO] 10.244.2.187:45050 - 36636 "PTR IN 10.0.96.10.in-addr.arpa. udp 41 false 512" NOERROR qr,aa,rd 116 0.000463271s
[INFO] 10.244.2.187:35271 - 55927 "A IN motioneye.home.svc.cluster.local.cert-manager.svc.cluster.local. udp 81 false 512" NXDOMAIN qr,aa,rd 174 0.000457012s
[INFO] 10.244.2.187:35271 - 56657 "AAAA IN motioneye.home.svc.cluster.local.cert-manager.svc.cluster.local. udp 81 false 512" NXDOMAIN qr,aa,rd 174 0.000653954s
[INFO] 10.244.2.187:46899 - 29479 "A IN motioneye.home.svc.cluster.local.svc.cluster.local. udp 68 false 512" NXDOMAIN qr,aa,rd 161 0.000474049s
[INFO] 10.244.2.187:46899 - 30158 "AAAA IN motioneye.home.svc.cluster.local.svc.cluster.local. udp 68 false 512" NXDOMAIN qr,aa,rd 161 0.000671509s
This is happening to all services, not just motioneye. The IP that it is resolving to is the LAN IP address of my Ingress controller. I'm not sure why.
MotionEye service:
apiVersion: v1
kind: Service
metadata:
labels:
app: motioneye
name: motioneye
namespace: home
spec:
externalTrafficPolicy: Local
ports:
- name: motioneye-http
protocol: TCP
port: YYYY
- name: motioneye-http2
protocol: TCP
port: XX
targetPort: YYYY
selector:
app: motioneye
Service for my Ingress controller:
apiVersion: v1
kind: Service
metadata:
labels:
app: ingress-nginx
name: quickstart-ingress-nginx-controller
namespace: cert-manager
spec:
ports:
- name: nginx-http
protocol: TCP
port: 80
- name: nginx-https
protocol: TCP
port: 443
loadBalancerIP: 10.7.9.10
type: LoadBalancer
Kube System namespace:
$ kbc get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcd69978-9xj4r 1/1 Running 0 9m47s
coredns-78fcd69978-wctr7 1/1 Running 0 9m47s
etcd-k-m-001 1/1 Running 57 (2d2h ago) 252d
kube-apiserver-k-m-001 1/1 Running 48 (2d2h ago) 252d
kube-controller-manager-k-m-001 1/1 Running 29 (2d2h ago) 252d
kube-flannel-ds-bgdgz 1/1 Running 2 (2d2h ago) 2d4h
kube-flannel-ds-cxhpv 1/1 Running 1 (2d2h ago) 2d4h
kube-flannel-ds-lh4qb 1/1 Running 2 (2d2h ago) 2d4h
kube-flannel-ds-r2x6w 1/1 Running 2 (2d2h ago) 2d4h
kube-flannel-ds-sfwvs 1/1 Running 2 (2d2h ago) 2d4h
kube-flannel-ds-wzj6r 0/1 Pending 0 2d2h
kube-proxy-9xxkm 1/1 Running 15 (2d2h ago) 171d
kube-proxy-kfcql 1/1 Running 23 (2d2h ago) 252d
kube-proxy-l886v 1/1 Running 12 (66d ago) 171d
kube-proxy-nt4dp 1/1 Running 30 (2d2h ago) 252d
kube-proxy-nvk5b 1/1 Running 22 (2d2h ago) 252d
kube-proxy-t55sz 1/1 Running 10 (2d2h ago) 171d
kube-scheduler-k-m-001 1/1 Running 43 (2d2h ago) 252d
metrics-server-5cd859f5c-jzff7 1/1 Running 155 (2d2h ago) 249d
I'm using Metal-LB for the load balancer.
Any help is appreciated!
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|