'Readiness probe issues on aks with helm elasticsearch installation
i am trying to install via helm elasticsearch on an azure aks this is the helm repo command :
helm repo add elastic https://helm.elastic.co
this is my values.yaml :
clusterName: "elasticsearch"
nodeGroup: "master"
# Elasticsearch roles that will be applied to this nodeGroup
# These will be set as environment variables. E.g. node.master=true
roles:
master: "true"
ingest: "true"
data: "true"
#Kubernetes replica count for the StatefulSet
replicas: 1
clusterHealthCheckParams: 'wait_for_status=yellow&timeout=1s'
minimumMasterNodes: 1
# Allows you to add any config files in /usr/share/elasticsearch/config/
# such as elasticsearch.yml
esConfig:
elasticsearch.yml: |
xpack.security.enabled: false
image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.12.0"
imagePullPolicy: "IfNotPresent"
esJavaOpts: "-Xms512m -Xmx512m"
resources:
requests:
cpu: "1000m"
memory: "4Gi"
limits:
cpu: "1000m"
memory: "4Gi"
networkHost: "0.0.0.0"
volumeClaimTemplate:
accessModes:
- ReadWriteMany
storageClassName: elk-azurefile-sc
resources:
requests:
storage: 10Gi
persistence:
enabled: true
labels:
# Add default labels for the volumeClaimTemplate of the StatefulSet
enabled: false
annotations: {}
protocol: http
httpPort: 9200
transportPort: 9300
sysctlVmMaxMapCount: 262144
note the :
clusterHealthCheckParams: 'wait_for_status=yellow&timeout=1s'
it has created a headless and a cluster-ip services successfully.wh
when I try to install it the pods go up the get an error :
Readiness probe failed: Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=yellow&timeout=1s" )
and then
Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: container is in CONTAINER_EXITED state
full kubectl describe pods elasticsearch-master-0:
Name: elasticsearch-master-0
Namespace: elk-stack
Priority: 0
Node: aks-large-26770588-vmss000000/10.224.0.6
Start Time: Sat, 14 May 2022 12:21:56 +0300
Labels: app=elasticsearch-master
chart=elasticsearch
controller-revision-hash=elasticsearch-master-79dc798b86
release=elasticsearch
statefulset.kubernetes.io/pod-name=elasticsearch-master-0
Annotations: configchecksum: d9df7552a0a018b2f9f12fc4e9f161c978aa773de39ccc7755dde7ddbe2ba92
Status: Running
IP: 10.244.47.6
IPs:
IP: 10.244.47.6
Controlled By: StatefulSet/elasticsearch-master
Init Containers:
configure-sysctl:
Container ID: containerd://381ad5812106422583bef9d2a22fd20a859f8ea2b7f447ff769a78cf6f858643
Image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0
Image ID: docker.elastic.co/elasticsearch/elasticsearch@sha256:4999c5f75c1d0d69754902d3975dd36875cc2eb4a06d7fdceaa8ec0e71a81dfa
Port: <none>
Host Port: <none>
Command:
sysctl
-w
vm.max_map_count=262144
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 14 May 2022 12:21:56 +0300
Finished: Sat, 14 May 2022 12:21:56 +0300
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6mz7h (ro)
Containers:
elasticsearch:
Container ID: containerd://a1f0c38c60f26368bc5cdbde890cb67eb0fb5d8e345e04f7496740124d830c17
Image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0
Image ID: docker.elastic.co/elasticsearch/elasticsearch@sha256:4999c5f75c1d0d69754902d3975dd36875cc2eb4a06d7fdceaa8ec0e71a81dfa
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 14 May 2022 12:34:36 +0300
Finished: Sat, 14 May 2022 12:34:53 +0300
Ready: False
Restart Count: 7
Limits:
cpu: 1
memory: 4Gi
Requests:
cpu: 1
memory: 4Gi
Readiness: exec [bash -c set -e
# If the node is starting up wait for the cluster to be ready (request params: "wait_for_status=yellow&timeout=1s" )
# Once it has started only check that the node itself is responding
START_FILE=/tmp/.es_start_file
# Disable nss cache to avoid filling dentry cache when calling curl
# This is required with Elasticsearch Docker using nss < 3.52
export NSS_SDB_USE_CACHE=no
http () {
local path="${1}"
local args="${2}"
set -- -XGET -s
if [ "$args" != "" ]; then
set -- "$@" $args
fi
if [ -n "${ELASTIC_PASSWORD}" ]; then
set -- "$@" -u "elastic:${ELASTIC_PASSWORD}"
fi
curl --output /dev/null -k "$@" "http://127.0.0.1:9200${path}"
}
if [ -f "${START_FILE}" ]; then
echo 'Elasticsearch is already running, lets check the node is healthy'
HTTP_CODE=$(http "/" "-w %{http_code}")
RC=$?
if [[ ${RC} -ne 0 ]]; then
echo "curl --output /dev/null -k -XGET -s -w '%{http_code}' \${BASIC_AUTH} http://127.0.0.1:9200/ failed with RC ${RC}"
exit ${RC}
fi
# ready if HTTP code 200, 503 is tolerable if ES version is 6.x
if [[ ${HTTP_CODE} == "200" ]]; then
exit 0
elif [[ ${HTTP_CODE} == "503" && "7" == "6" ]]; then
exit 0
else
echo "curl --output /dev/null -k -XGET -s -w '%{http_code}' \${BASIC_AUTH} http://127.0.0.1:9200/ failed with HTTP code ${HTTP_CODE}"
exit 1
fi
else
echo 'Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=yellow&timeout=1s" )'
if http "/_cluster/health?wait_for_status=yellow&timeout=1s" "--fail" ; then
touch ${START_FILE}
exit 0
else
echo 'Cluster is not yet ready (request params: "wait_for_status=yellow&timeout=1s" )'
exit 1
fi
fi
] delay=10s timeout=5s period=10s #success=3 #failure=3
Environment:
node.name: elasticsearch-master-0 (v1:metadata.name)
cluster.initial_master_nodes: elasticsearch-master-0,
discovery.seed_hosts: elasticsearch-master-headless
cluster.name: elasticsearch
network.host: 0.0.0.0
cluster.deprecation_indexing.enabled: false
ES_JAVA_OPTS: -Xms512m -Xmx512m
node.data: true
node.ingest: true
node.master: true
node.ml: true
node.remote_cluster_client: true
Mounts:
/usr/share/elasticsearch/config/elasticsearch.yml from esconfig (rw,path="elasticsearch.yml")
/usr/share/elasticsearch/data from elasticsearch-master (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6mz7h (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
elasticsearch-master:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: elasticsearch-master-elasticsearch-master-0
ReadOnly: false
esconfig:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: elasticsearch-master-config
Optional: false
kube-api-access-6mz7h:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 17m default-scheduler Successfully assigned elk-stack/elasticsearch-master-0 to aks-large-26770588-vmss000000
Normal Pulled 17m kubelet Container image "docker.elastic.co/elasticsearch/elasticsearch:7.12.0" already present on machine
Normal Created 17m kubelet Created container configure-sysctl
Normal Started 17m kubelet Started container configure-sysctl
Warning Unhealthy 16m (x2 over 16m) kubelet Readiness probe failed: Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=yellow&timeout=1s" )
Cluster is not yet ready (request params: "wait_for_status=yellow&timeout=1s" )
Normal Created 15m (x4 over 17m) kubelet Created container elasticsearch
Normal Started 15m (x4 over 17m) kubelet Started container elasticsearch
Warning Unhealthy 15m kubelet Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: container is in CONTAINER_EXITED state
Normal Pulled 14m (x5 over 17m) kubelet Container image "docker.elastic.co/elasticsearch/elasticsearch:7.12.0" already present on machine
Warning BackOff 119s (x60 over 16m) kubelet Back-off restarting failed container
logs of container:
{"type": "server", "timestamp": "2022-05-14T16:22:42,172Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elasticsearch-master-0, elasticsearch-master-1, elasticsearch-master-2] to bootstrap a cluster: have discovered [{elasticsearch-master-0}{hchEvQdFQrW0sNFcB1Trew}{QsHcqMzFT7ibSJ1iDPK1pQ}{10.244.47.18}{10.244.47.18:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{hchEvQdFQrW0sNFcB1Trew}{QsHcqMzFT7ibSJ1iDPK1pQ}{10.244.47.18}{10.244.47.18:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2022-05-14T16:22:52,173Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elasticsearch-master-0, elasticsearch-master-1, elasticsearch-master-2] to bootstrap a cluster: have discovered [{elasticsearch-master-0}{hchEvQdFQrW0sNFcB1Trew}{QsHcqMzFT7ibSJ1iDPK1pQ}{10.244.47.18}{10.244.47.18:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{hchEvQdFQrW0sNFcB1Trew}{QsHcqMzFT7ibSJ1iDPK1pQ}{10.244.47.18}{10.244.47.18:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2022-05-14T16:22:53,476Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "path: /_cluster/health, params: {wait_for_status=green, timeout=200s}",
"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.3.jar:7.17.3]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
"at java.lang.Thread.run(Thread.java:833) [?:?]"] }
{"type": "server", "timestamp": "2022-05-14T16:23:02,175Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elasticsearch-master-0, elasticsearch-master-1, elasticsearch-master-2] to bootstrap a cluster: have discovered [{elasticsearch-master-0}{hchEvQdFQrW0sNFcB1Trew}{QsHcqMzFT7ibSJ1iDPK1pQ}{10.244.47.18}{10.244.47.18:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{hchEvQdFQrW0sNFcB1Trew}{QsHcqMzFT7ibSJ1iDPK1pQ}{10.244.47.18}{10.244.47.18:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2022-05-14T16:23:12,176Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elasticsearch-master-0, elasticsearch-master-1, elasticsearch-master-2] to bootstrap a cluster: have discovered [{elasticsearch-master-0}{hchEvQdFQrW0sNFcB1Trew}{QsHcqMzFT7ibSJ1iDPK1pQ}{10.244.47.18}{10.244.47.18:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{hchEvQdFQrW0sNFcB1Trew}{QsHcqMzFT7ibSJ1iDPK1pQ}{10.244.47.18}{10.244.47.18:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2022-05-14T16:23:22,177Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elasticsearch-master-0, elasticsearch-master-1, elasticsearch-master-2] to bootstrap a cluster: have discovered [{elasticsearch-master-0}{hchEvQdFQrW0sNFcB1Trew}{QsHcqMzFT7ibSJ1iDPK1pQ}{10.244.47.18}{10.244.47.18:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{hchEvQdFQrW0sNFcB1Trew}{QsHcqMzFT7ibSJ1iDPK1pQ}{10.244.47.18}{10.244.47.18:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2022-05-14T16:23:32,178Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elasticsearch-master-0, elasticsearch-master-1, elasticsearch-master-2] to bootstrap a cluster: have discovered [{elasticsearch-master-0}{hchEvQdFQrW0sNFcB1Trew}{QsHcqMzFT7ibSJ1iDPK1pQ}{10.244.47.18}{10.244.47.18:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{hchEvQdFQrW0sNFcB1Trew}{QsHcqMzFT7ibSJ1iDPK1pQ}{10.244.47.18}{10.244.47.18:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2022-05-14T16:23:42,180Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elasticsearch-master-0, elasticsearch-master-1, elasticsearch-master-2] to bootstrap a cluster: have discovered [{elasticsearch-master-0}{hchEvQdFQrW0sNFcB1Trew}{QsHcqMzFT7ibSJ1iDPK1pQ}{10.244.47.18}{10.244.47.18:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{hchEvQdFQrW0sNFcB1Trew}{QsHcqMzFT7ibSJ1iDPK1pQ}{10.244.47.18}{10.244.47.18:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
elasticsearchSources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|