Kubernetes Nginx ingress trying to get the wrong SSL certificate

Question

I have a strange case. Context:

At the very first, the client was using our domain for their store, the URL was something like somestore.eu.mycompany.com

Then, the client upgraded to a custom domain, other clients did this without any problem.

We deleted the whole namespace with the old subdomain and created a new one with the domain.

The root domain works flawlessly, without SSL certificate issues. However the staging subdomain works sometimes, sometimes without an SSL certificate issue, sometime with this error:

$ curl -vI https://staging.somestore.com/
*   Trying 35.102.186.11:443...
* TCP_NODELAY set
* Connected to staging.somestore.com (35.102.186.11) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.eu.mycompany.com
*  start date: Mar 13 09:29:46 2022 GMT
*  expire date: Jun 11 09:29:45 2022 GMT
*  subjectAltName does not match staging.somestore.com
* SSL: no alternative certificate subject name matches target host name 'staging.somestore.com'

Looking at the logs I can see that nginx-ingress still trying to get the old certificate

kubectl logs -f -n ingress-nginx nginx-ingress-controller-55f88544bf-dk7ht | grep my-namespace

SSL certificate "my-namespace/tls-cert" does not contain a Common Name or Subject Alternative Name for server "somestore.eu.mycompany.com": x509: certificate is valid for somestore.com, staging.somestore.com, www.somestore.com, not somestore.eu.mycompany.com

Why Kubernetes's nginx-ingress still trying to get the old certificate?

Answer 1

1. make sure you have the correct configmaps, secrets or other configuration in your cluster (E.g. where SSL certs are stored). The desired config must be present, the deprecated must be dumped.
2. perform a rollout restart on your deployment. ( E.g. if nginx-ingress is the name of the deployment in the ingress namespace, do this: kubectl rollout restart -n ingress deploy/nginx-ingress )