'Opensearch service not reachable to App services on EKS Cluster behind NGINX Ingress
I am trying to setup my application and opensearch application behind NGINX ingress on EKS as shown below.
On NGINX Ingress, I setup the routing as below which is working fine:
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-opensearch
namespace: os-dev
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
ingressClassName: nginx
tls:
- hosts:
- acdfdd7b4bsdfe3ed1ae568bc892743e5-022f05b723bf7c62.elb.us-east-2.amazonaws.com
secretName: tls-for-dashboards-key-pair
rules:
- host: acdfdd7b4bsdfe3ed1ae568bc892743e5-022f05b723bf7c62.elb.us-east-2.amazonaws.com
http:
paths:
- path: /os(/|$)(.*)
pathType: Prefix
backend:
service:
name: opensearch-cluster-master
port:
number: 9200
The above one is working from browser with https://acdfdd7b4bsdfe3ed1ae568bc892743e5-022f05b723bf7c62.elb.us-east-2.amazonaws.com/os.
The service status shows everything looks good:
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cert-manager cert-manager ClusterIP 10.100.230.233 <none> 9402/TCP 6d8h
cert-manager cert-manager-webhook ClusterIP 10.100.240.163 <none> 443/TCP 6d8h
default kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 6d11h
ingress ing-ingress-nginx-controller LoadBalancer 10.100.69.69 acdfdd7b4bsdfe3ed1ae568bc892743e5-022f05b723bf7c62.elb.us-east-2.amazonaws.com 80:30947/TCP,443:32576/TCP 6d
ingress ing-ingress-nginx-controller-admission ClusterIP 10.100.211.227 <none> 443/TCP 6d
kube-system kube-dns ClusterIP 10.100.0.10 <none> 53/UDP,53/TCP 6d11h
os-dev dashboards-opensearch-dashboards ClusterIP 10.100.85.97 <none> 5601/TCP 5d5h
os-dev opensearch-cluster-master ClusterIP 10.100.245.3 <none> 9200/TCP,9300/TCP 5d5h
os-dev opensearch-cluster-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 5d5h
On the application side, whenever the OpenSearch endpoint is given as opensearch-cluster-master:9200, the application is failing to connect.
However, when I tested from container to ping the OpenSearch container, the ping is working fine but the telnet to port is failing.
Any leads on this really appreciated.
Pod status:
[root@ip-172-31-24-175 matilda_deployment]# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cert-manager cert-manager-847544bbd-xbbnj 1/1 Running 0 6d8h
cert-manager cert-manager-cainjector-5c747645bf-s4x7k 1/1 Running 0 6d8h
cert-manager cert-manager-webhook-f588b48b8-rlhq4 1/1 Running 0 6d8h
ingress ing-ingress-nginx-controller-64b96466c4-lgz4q 1/1 Running 0 6d
kube-system aws-node-jgz25 1/1 Running 0 6d10h
kube-system aws-node-vf6bl 1/1 Running 0 6d10h
kube-system coredns-f47955f89-9kl8d 1/1 Running 0 6d11h
kube-system coredns-f47955f89-jr5xf 1/1 Running 0 6d11h
kube-system kube-proxy-krlrh 1/1 Running 0 6d10h
kube-system kube-proxy-mmlkc 1/1 Running 0 6d10h
os-dev app-6c498bd64c-z9hn7 0/1 CrashLoopBackOff 31 139m
os-dev dashboards-opensearch-dashboards-655fc6ff46-6gfm4 1/1 Running 0 5d5h
os-dev opensearch-cluster-master-0 1/1 Running 0 5d6h
os-dev opensearch-cluster-master-1 1/1 Running 0 5d6h
os-dev opensearch-cluster-master-2 1/1 Running 0 5d6h
Thank you in advance.
[UPDATE]: this issue is more application specific where the certificates are generated with wrong domain. Hence closing the issue.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|