Issue pulling secret for NGINX Ingress Controller with NLB in EKS

Question

Trying to provision NGINX ingress controller with NLB in EKS. Getting CrashLoopBackOff for ingress-nginx-admission-create and ingress-nginx-admission-patch.

It's worth mentioning that this is a private EKS cluster without internet access and I'm pulling the docker images successfully from ECR. Also, I am using a secondary VPC CIDR to allocate pod IPs.

I am following this documentation:

https://kubernetes.github.io/ingress-nginx/deploy/#aws

https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/aws/deploy.yaml

The following resources are created:

namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created

NAMESPACE       NAME                                        READY   STATUS              RESTARTS   AGE
ingress-nginx   ingress-nginx-admission-create-lws8l        0/1     CrashLoopBackOff    4          4m18s
ingress-nginx   ingress-nginx-admission-patch-g5s8w         0/1     CrashLoopBackOff    4          4m18s
ingress-nginx   ingress-nginx-controller-79c469cd9f-wqmhn   0/1     ContainerCreating   0          4m18s
kube-system     aws-node-4g2h2                              1/1     Running             0          29h
kube-system     aws-node-r65xb                              1/1     Running             0          29h
kube-system     aws-node-spfzj                              1/1     Running             0          29h
kube-system     coredns-65ccb76b7c-97xcv                    1/1     Running             0          31h
kube-system     coredns-65ccb76b7c-bck8f                    1/1     Running             0          31h
kube-system     kube-proxy-jnfjd                            1/1     Running             0          29h
kube-system     kube-proxy-smc88                            1/1     Running             0          29h
kube-system     kube-proxy-v6hjp                            1/1     Running             0          29h

One of the issues seems to be getting the secret and I don't understand why.

{"err":"Get \"https://172.20.0.1:443/api/v1/namespaces/ingress-nginx/secrets/ingress-nginx-admission\": dial tcp 172.20.0.1:443: i/o timeout","level":"fatal","msg":"error getting secret","source":"k8s/k8s.go:232","time":"2022-03-11T23:56:16Z"}

The other thing is that the NLB is not being created.

sh-4.2$ kubectl get svc -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   172.20.76.158    <pending>     80:31054/TCP,443:32308/TCP   7m43s
ingress-nginx-controller-admission   ClusterIP      172.20.240.214   <none>        443/TCP                      7m43s

sh-4.2$ kubectl get -A ValidatingWebhookConfiguration
NAME                              WEBHOOKS   AGE
ingress-nginx-admission           1          12m
vpc-resource-validating-webhook   1          31h

Here is the ingress-nginx-admission-create-xw4rz pod.

Name:         ingress-nginx-admission-create-xw4rz
Namespace:    ingress-nginx
Priority:     0
Node:         ip-10-51-80-103.eu-west-2.compute.internal/10.51.80.103
Start Time:   Sat, 12 Mar 2022 14:38:38 +0000
Labels:       app.kubernetes.io/component=admission-webhook
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/version=1.1.1
              controller-uid=e6d21979-55bf-4b44-aa41-6462b9923806
              helm.sh/chart=ingress-nginx-4.0.15
              job-name=ingress-nginx-admission-create
Annotations:  kubernetes.io/psp: eks.privileged
Status:       Running
IP:           10.0.25.33
IPs:
  IP:           10.0.25.33
Controlled By:  Job/ingress-nginx-admission-create
Containers:
  create:
    Container ID:  docker://9a311111111111111111111111111111111111111111111111111    Image:         11111111111111.dkr.ecr.eu-west-2.amazonaws.com/certgen:latest
    Image ID:      docker-pullable://11111111111111.dkr.ecr.eu-west-2.amazonaws.com/certgen@sha256:7831111111111111111111111111111
    Port:          <none>
    Host Port:     <none>
    Args:
      create
      --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
      --namespace=$(POD_NAMESPACE)
      --secret-name=ingress-nginx-admission
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sat, 12 Mar 2022 14:42:05 +0000
      Finished:     Sat, 12 Mar 2022 14:42:35 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sat, 12 Mar 2022 14:40:52 +0000
      Finished:     Sat, 12 Mar 2022 14:41:22 +0000
    Ready:          False
    Restart Count:  4
    Environment:
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rrlnd (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-rrlnd:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  3m59s                default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-admission-create-xw4rz to ip-10-51-80-103.eu-west-2.compute.internal
  Normal   Pulled     32s (x5 over 3m58s)  kubelet            Container image "111111111111111.dkr.ecr.eu-west-2.amazonaws.com/certgen:latest" already present on machine
  Normal   Created    32s (x5 over 3m58s)  kubelet            Created container create
  Normal   Started    32s (x5 over 3m58s)  kubelet            Started container create
  Warning  BackOff    2s (x7 over 2m56s)   kubelet            Back-off restarting failed container

And here is the ingress-nginx-controller-79c469cd9f-ft76q. I can see a failed mount, but I understand that "ingress-nginx-controller" is created after the 2 pods above run without errors.

sh-4.2$ kubectl describe pod ingress-nginx-controller-79c469cd9f-ft76q -n ingress-nginx
Name:           ingress-nginx-controller-79c469cd9f-ft76q
Namespace:      ingress-nginx
Priority:       0
Node:           ip-10-51-80-6.eu-west-2.compute.internal/10.51.80.6
Start Time:     Sat, 12 Mar 2022 14:38:38 +0000
Labels:         app.kubernetes.io/component=controller
                app.kubernetes.io/instance=ingress-nginx
                app.kubernetes.io/name=ingress-nginx
                pod-template-hash=79c469cd9f
Annotations:    kubernetes.io/psp: eks.privileged
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/ingress-nginx-controller-79c469cd9f
Containers:
  controller:
    Container ID:
    Image:         111111111111111.dkr.ecr.eu-west-2.amazonaws.com/nginx-controller:latest
    Image ID:
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-79c469cd9f-ft76q (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j8v4c (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-j8v4c:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                    From               Message
  ----     ------       ----                   ----               -------
  Normal   Scheduled    8m46s                  default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-controller-79c469cd9f-ft76q to ip-10-51-80-6.eu-west-2.compute.internal
  Warning  FailedMount  6m43s                  kubelet            Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[kube-api-access-j8v4c webhook-cert]: timed out waiting for the condition
  Warning  FailedMount  2m14s (x2 over 4m29s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[webhook-cert kube-api-access-j8v4c]: timed out waiting for the condition
  Warning  FailedMount  32s (x12 over 8m46s)   kubelet            MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found