AWS Load Balancer Controller: unable to recognize "": no matches for kind "IngressClass" in version "networking.k8s.io/v1"

Question

I am trying to install the AWS Load Balancer Controller in my EKS cluster, which use to work in the current version of my EKS cluster. While doing a fresh deployment (showing a colleague how easy it is...doh!), I am now getting an error. Details below:

EKS Cluster: 1.18 (this has not changed)

helm: v3.8.1

kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"

helm repo add eks https://aws.github.io/eks-charts

helm upgrade \
  --install aws-load-balancer-controller eks/aws-load-balancer-controller \
  --set clusterName=EKSCluster-d7lBmlStzxdg \
  --set image.tag=v2.3.1 \
  --set serviceAccount.create=true \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set serviceAccount.annotations.eks\.amazonaws\.com/role-arn=arn:aws:iam::xxxxxxxxxxxx:role/aws-load-balancer-controller-EKSCluster-xxxxxxxx \
  --set rbac.create=true \
  --set region=us-west-2 \
  --set vpcId=vpc-xxxxxxxxxxxxx 
   --namespace kube-system

This is my ingress.yaml:

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/certificate-arn: {{ .Values.endpoint.acmTlsArn }}
    alb.ingress.kubernetes.io/wafv2-acl-arn: {{ .Values.endpoint.wafv2Arn }}
    alb.ingress.kubernetes.io/healthcheck-port: traffic-port
    external-dns.alpha.kubernetes.io/hostname: {{ .Values.endpoint.fqdn }}
    kots.io/app-slug: stoplight-platform
  generation: 1
  labels:
    app.kubernetes.io/component: platform-ingress
    app.kubernetes.io/instance: platform
    app.kubernetes.io/name: stoplight
  name: platform-platform-ingress
  namespace: {{ .Values.app.namespace }}
spec:
  rules:
  - host: {{ .Values.endpoint.fqdn }}
    http:
      paths:
      - backend:
          serviceName: platform-nginx
          servicePort: 80

Every-time I get this error:

Release "aws-load-balancer-controller" does not exist. Installing it now.
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "IngressClass" in version "networking.k8s.io/v1"

I thought it was because the newer version of AWS Load Balancer Controller (v2.4.0) requires k8s v1.19+, so I set the image tag to use: --set image.tag=v2.3.1 which is supposed to be compatible with k8s 1.18. But same error.

Any help would be appreciated, kinda still a k8s newb. Again, all this use to work without errors, but something has changed and I do not know what.

Update: Running with --debug i get this:

customresourcedefinition.apiextensions.k8s.io/ingressclassparams.elbv2.k8s.aws configured
customresourcedefinition.apiextensions.k8s.io/targetgroupbindings.elbv2.k8s.aws configured

"eks" already exists with the same configuration, skipping

/opt/homebrew/bin/helm upgrade --install aws-load-balancer-controller eks/aws-load-balancer-controller --set clusterName=EKSCluster-d7lBmlStzxdg --set image.tag=v2.3.1 --set serviceAccount.create=true --set serviceAccount.name=aws-load-balancer-controller --set serviceAccount.annotations.eks\.amazonaws\.com/role-arn=arn:aws:iam::xxxxxxxxxxxxx:role/aws-load-balancer-controller-EKSCluster-d7lBmlStzxdg --set rbac.create=true --set region=us-west-2 --set vpcId=vpc-xxxxxxxxxxxxxxx --namespace kube-system
history.go:56: [debug] getting history for release aws-load-balancer-controller
Release "aws-load-balancer-controller" does not exist. Installing it now.
install.go:178: [debug] Original chart version: ""
install.go:195: [debug] CHART PATH: /Users/conlann/Library/Caches/helm/repository/aws-load-balancer-controller-1.4.1.tgz

client.go:128: [debug] creating 2 resource(s)
install.go:151: [debug] CRD ingressclassparams.elbv2.k8s.aws is already present. Skipping.
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "IngressClass" in version "networking.k8s.io/v1"
helm.go:84: [debug] unable to recognize "": no matches for kind "IngressClass" in version "networking.k8s.io/v1"
unable to build kubernetes objects from release manifest
helm.sh/helm/v3/pkg/action.(*Install).RunWithContext
        helm.sh/helm/v3/pkg/action/install.go:277
main.runInstall
        helm.sh/helm/v3/cmd/helm/install.go:264
main.newUpgradeCmd.func2
        helm.sh/helm/v3/cmd/helm/upgrade.go:120
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/[email protected]/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/[email protected]/command.go:974
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/[email protected]/command.go:902
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
        runtime/proc.go:255
runtime.goexit
        runtime/asm_arm64.s:1133
make: *** [controller-alb-ingress] Error 

Answer 1

Looks like you are using an older manifest (prior to K8s 1.18) that uses the ingress class annotation:

kubernetes.io/ingress.class: alb

In the newer Ingress resource definition (K8s 1.18+) you specify it as part of the spec in the resource:

Example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: alb
  rules:
  - http:
  ...

A workaround is to just define a default IngressClass:

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  labels:
    app.kubernetes.io/component: controller
  name: nginx-example
  annotations:
    ingressclass.kubernetes.io/is-default-class: "true"
spec:
  controller: k8s.io/ingress-nginx

Lots of info in the K8s ingress docs.

??