'kubectl proxy is unable to reach pod services: steps I should attempt to investigate/diagnose?

I have a baremetal k8s cluster (1.23.5, calico as CNI), and after some uptime I encountered that kubectl proxy is unable to reach pod services:

proxying:

kubectl proxy --port=8008
Starting to serve on 127.0.0.1:8008

requesting:

curl http://localhost:8008/api/v1/namespaces/emz/pods/nginx-6b78d5b64c-km5lr:80/proxy/
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "error trying to reach service: dial tcp 10.200.3.21:80: i/o timeout",
  "reason": "ServiceUnavailable",
  "code": 503
}

No matter what pod, service or namespace - the result is the same. Pods are running without errors, pods can ping each other among different namespaces, DNS resolve is working inside pods, and api objects are browseable inside k8s api (I omit most of the sample output because it's fully normal and cumbersome):

> curl http://localhost:8008/api/v1/namespaces/emz/pods/nginx-6b78d5b64c-km5lr
{
  "kind": "Pod",
  "apiVersion": "v1",
  "metadata": {
    "name": "nginx-6b78d5b64c-km5lr",
    "generateName": "nginx-6b78d5b64c-",
    "namespace": "emz",
    "uid": "819b94de-7aa6-441a-a40e-805a0f79e9a8",
    "resourceVersion": "8070751",
[...]
    "containers": [
      {
        "name": "nginx",
        "image": "nginx",
        "ports": [
          {
            "name": "nginx",
            "containerPort": 80,
            "protocol": "TCP"
          }
        ],
[...]
  "status": {
    "phase": "Running",
[...]
    "podIP": "10.200.3.21",
    "podIPs": [
      {
        "ip": "10.200.3.21"
      }
    ],

What is even more weird - is that kubectl port-forward is able to reach said services:

port-forward:

> kubectl port-forward -n emz nginx-6b78d5b64c-km5lr 8008:80
Forwarding from 127.0.0.1:8008 -> 80
Forwarding from [::1]:8008 -> 80

request:

> curl 127.0.0.1:8008
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Can anyone please point me to the direction on how to diagnose/investigate the reason for the kubectl proxy to stuck in these timeouts ? Cluster seems to be fully operating besides that.

Thanks.

Update: Seems like the reason of this, direct or relatedб may be in the fact that all of the three control nodes lost connectivity via the internal pod network to other pods. I have drained and rebooted one node, but this didn't help at all.

Update 2: In fact it did help to restore the connectivity from this node to the indernal network. Weird, but calico pods/deployments were not complaining about loss of connectivity.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source

'kubectl proxy is unable to reach pod services: steps I should attempt to investigate/diagnose?

Sources

Related Questions