Let's Encrypt kubernetes Ingress Controller issuing Fake Certificate

Question

Not Sure why I'm getting Fake certificate, even the certificate is properly issued by Let's Encrypt using certmanager

The setup is running on the Alibaba Cloud ECS console, where one Kube-master and one cube-minion form a Kubernetes cluster.

Service Details

root@kube-master:~# kubectl get svc 
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   3h20m
my-nginx     ClusterIP   10.101.150.247   <none>        80/TCP    77m

Pod Details

root@kube-master:~# kubectl get pods --show-labels
NAME                        READY   STATUS    RESTARTS   AGE   LABELS
my-nginx-6cc48cd8db-n6scm   1/1     Running   0          46s   app=my-nginx,pod-template-hash=6cc48cd8db

Helm Cert-manager deployed

root@kube-master:~# helm ls 
NAME            REVISION    UPDATED                     STATUS      CHART               APP VERSION NAMESPACE  
cert-manager    1           Tue Mar 12 15:29:21 2019    DEPLOYED    cert-manager-v0.5.2 v0.5.2      kube-system
kindred-garfish 1           Tue Mar 12 17:03:41 2019    DEPLOYED    nginx-ingress-1.3.1 0.22.0      kube-system

Certificate Issued Properly

root@kube-master:~# kubectl describe certs 
Name:         tls-prod-cert
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2019-03-12T10:26:58Z
  Generation:          2
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  nginx-ingress-prod
    UID:                   5ab11929-44b1-11e9-b431-00163e005d19
  Resource Version:        17687
  Self Link:               /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/tls-prod-cert
  UID:                     5dad4740-44b1-11e9-b431-00163e005d19
Spec:
  Acme:
    Config:
      Domains:
        zariga.com
      Http 01:
        Ingress:        
        Ingress Class:  nginx
  Dns Names:
    zariga.com
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-prod
  Secret Name:  tls-prod-cert
Status:
  Acme:
    Order:
      URL:  https://acme-v02.api.letsencrypt.org/acme/order/53135536/352104603
  Conditions:
    Last Transition Time:  2019-03-12T10:27:00Z
    Message:               Order validated
    Reason:                OrderValidated
    Status:                False
    Type:                  ValidateFailed
    Last Transition Time:  <nil>
    Message:               Certificate issued successfully
    Reason:                CertIssued
    Status:                True
    Type:                  Ready
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  CreateOrder   27s   cert-manager  Created new ACME order, attempting validation...
  Normal  IssueCert     27s   cert-manager  Issuing certificate...
  Normal  CertObtained  25s   cert-manager  Obtained certificate from ACME server
  Normal  CertIssued    25s   cert-manager  Certificate issued successfully

Ingress Details

root@kube-master:~# kubectl describe ingress
Name:             nginx-ingress-prod
Namespace:        default
Address:          
Default backend:  my-nginx:80 (192.168.123.202:80)
TLS:
  tls-prod-cert terminates zariga.com
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     *     my-nginx:80 (192.168.123.202:80)
Annotations:
  kubernetes.io/ingress.class:        nginx
  kubernetes.io/tls-acme:             true
  certmanager.k8s.io/cluster-issuer:  letsencrypt-prod
Events:
  Type    Reason             Age    From                      Message
  ----    ------             ----   ----                      -------
  Normal  CREATE             7m13s  nginx-ingress-controller  Ingress default/nginx-ingress-prod
  Normal  CreateCertificate  7m8s   cert-manager              Successfully created Certificate "tls-prod-cert"
  Normal  UPDATE             6m57s  nginx-ingress-controller  Ingress default/nginx-ingress-prod

Letsencrypt Nginx Production Definition

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress-prod
  annotations:
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/tls-acme: 'true'
  labels:
    app: 'my-nginx'
spec:
  backend:
    serviceName: my-nginx
    servicePort: 80
  tls:
  - secretName: tls-prod-cert
    hosts:
    - zariga.com

Answer 1

Maybe would be helpful for someone experiencing similar issues. As for me, a forgot to specify hostname in Ingress yaml file for both rules and tls sections. After duplicating the hostname, it started responding with a proper certificate.

Example:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: test-web-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  tls:
  - hosts:
    - my.host.com                # <----
    secretName: tls-secret
  rules:
    - host: my.host.com          # <----
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              serviceName: my-nginx
              servicePort: 80

Answer 2

Sometimes it may happen if you are using the clusterissuer URL as staging URL.

Check the letsencrypt url set in you issuer.yaml or clusterissuer.yaml and change it to production url: https://acme-v02.api.letsencrypt.org/directory

I faced the same issue once and changing the url to production url solved it.

Also check that the ingress tls secrets you are using is right.

Actual cluster issuer should be something like for production :

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: dev-clusterissuer
spec:
  acme:
    email: [email protected]
    privateKeySecretRef:
      name: dev-clusterissuer
    server: https://acme-v02.api.letsencrypt.org/directory       # <----check this server URL it is for Prod and use this only
    solvers:
    - http01:
        ingress:
          class: nginx

If you are using server: https://acme-staging-v02.api.letsencrypt.org/directory you will face issue better replace it with server: https://acme-v02.api.letsencrypt.org/directory

Answer 3

If you're convinced that everything is set up correctly and it still doesn't work, try this.

Edit the deployment of your nginx-controller. Why? Because, if it doesn't find the secret in the namespace it's deployed in, the Nginx controller deploys it's own certificate (fake certificate). Not knowing this (I'm new to the game) cost me a few days of my life.

So, either change to the namespace where your Nginx Ingress controller is and get the name of the deployment, then:

kubectl edit deployment nginx-ingress-ingress-nginx-controller -n nginx-ingress

Or if there is only one deployment in that namespace you can just do

kubectl edit deployment

And you should be in edit mode for your nginx controller deployment. Look for the section: spec --> containers: --> args:

 spec:
  containers:
  - args:
    - /nginx-ingress-controller
    - --publish-service=$(POD_NAMESPACE)/nginx-ingress-ingress-nginx-controller
    - --election-id=ingress-controller-leader
    - --ingress-class=nginx
    - --configmap=$(POD_NAMESPACE)/nginx-ingress-ingress-nginx-controller
    - --validating-webhook=:8443
    - --validating-webhook-certificate=/usr/local/certificates/cert
    - --validating-webhook-key=/usr/local/certificates/key
    - --default-ssl-certificate=app-namespace/letsencrypt-cert-prod

You can add a default certificate to use if your nginx controller doesn't find one (as I have above), so it will search in a namespace for a secret by adding:

--default-ssl-certificate=your-cert-namespace/your-cert-secret

your-cert-namespace: The namespace where your certificate secret is your-cert-secret: The name of your certificate containing secret

Once you save and close your editor, it should be updated. Then check the logs of your cert manager pod:

kubectl logs cert-manager-xxxpodxx-abcdef -n cert-manager

To make sure that things are working as normal.

You probably won't have this issue if all your resources are deployed in the same namespace.

Answer 4

Important to note that the ClusterIssuer spec for solvers changed. For people using cer-manager>0.7.2, this comment saved me so much time: https://github.com/jetstack/cert-manager/issues/1650#issuecomment-518953464. Specially on how to configure the ClusterIssuer and Certificate.

Answer 5

For me, the issue was ingress class name, since I'm using microk8s, ingress class name is public:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: "[email protected]"
    privateKeySecretRef:
      name: letsencrypt-prod
    server: "https://acme-v02.api.letsencrypt.org/directory"
    solvers:
      - http01:
         ingress:
           class: public

Answer 6

In my case, the problem was accessing the domain at wrong port, my default https port wasn't 443 but 4443

Answer 7

For me, the issue was that I forgot to kubectl apply the secret (in my case 'tls-secret.yml'). When deploying K8S manually, such an error is rarely made. However, I'm using gitlab CICD to deploy applications, and I forgot to add - kubectl apply -f ./kube/secret to my .gitlab-ci.yml.