Using Location Blocks In Kubernetes Ingress Nginx server-snippet Causes 404

Question

I'm hoping someone can help me here because I'm stuck.

I'm moving over our nginx config from a traditional nginx/node server config whereby both nginx and node server are on the same machine.

In Kubernetes, the ingress controller (nginx) obviously lives in another container.

Where I'm getting stuck is reimplementing our rules that disable access logging for images and assets using location blocks.

Our configuration looks like

location ~* \.(?:jpg|jpeg|gif|png|ico|cur|mp4|ogg|ogv|webm|htc)$ {
  access_log off;
  expires 2M;
  add_header Cache-Control "public, max-age=5184000"; # 5184000 is 60 days
}

When I implement this same block in a server-snippet it matches, but all the assets throw a 404.

I did some Googling and found an answer that might explain why here https://stackoverflow.com/a/52711388/573616

but the suggested answer hints to use an if block instead of a location block because the location interferes with the proxy upstream, however, disabling access logs is not possible from inside the if block, only from a location context.

The rest of my ingress looks like (everything else is default)

real_ip_header X-Forwarded-For;
real_ip_recursive on;        
underscores_in_headers on;
gzip_types text/css application/x-javascript application/javascript application/json image/svg+xml;
client_max_body_size 5M;

proxy_buffers 8 16k;      
proxy_set_header X-Request-Start "t=${msec}";   
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_redirect off;

The images live at /images/ on the upstream server path.

So I'm back to trying to figure out how to get these location blocks working so I can actually disable the access logs for these images from a server-snippet

So can anyone tell me how to get the above location block to not throw 404's for assets in an ingress controller?

Answer 1

I'm assuming that your backend is serving your assets, so I think the problem is that your location {} block doesn't have an upstream like the regular paths defined in the nginx ingress.

There's a lot of lua code in the nginx.conf of your nginx-ingress-controller so it might take time to understand, but you can copy your nginx.conf locally:

$ kubectl cp nginx-ingress-controller-xxxxxxxxx-xxxxx:nginx.conf .

Check the location {} blocks that are defined for your current services and copy them in the bottom of your server-snippet location {} block like this:

I believe a server-snippet like this would do:

location ~* \.(?:jpg|jpeg|gif|png|ico|cur|mp4|ogg|ogv|webm|htc)$ {
  access_log off;
  expires 2M;
  add_header Cache-Control "public, max-age=5184000"; # 5184000 is 60 days
  <== add what you copied here
  set $namespace      "k8s-namespace";
  set $ingress_name   "ingress-name";
  set $service_name   "service-name";
  set $service_port   "80";
  set $location_path  "/images";
  ...
  ...
  ...
  proxy_http_version                      1.1;

  proxy_cookie_domain                     off;
  proxy_cookie_path                       off;

  # In case of errors try the next upstream server before returning an error
  proxy_next_upstream                     error timeout;
  proxy_next_upstream_tries               3;

  proxy_pass http://upstream_balancer;

  proxy_redirect                          off;
}

Answer 2

There is another approach to this. You can actually do the ingress way to handle this static content request(dynamically), without editing nginx.conf . Moreover nginx.conf is dynamically created/edited by the ingress-nginx controller every time you make changes in the ingress object. Making changes or mounting the nginx.conf file can create issues.

For the Static files, use the below snippet to redirect your request to the correct service.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sample-app-ingress
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/server-snippet: |
      expires 1M;
      add_header Cache-Control "public";
spec:
  ingressClassName: nginx
  rules:
  - host: xx.xx.com
      http:
        paths:
        - pathType: Prefix
          path: /status/.*
          backend:
            service:
              name: one-service
              port:
                number: 80
  - host: xx.xx.com
     http:
       paths:
        - pathType: Prefix
          path: /.*(js|css|gif|jpe?g|png)
          backend:
            service:
              name: static-app-service
              port:
                number: 80
  - host: xx.xx.com
      http:
        paths:
        - pathType: Prefix
          path: /
          backend:
            service:
              name: third-service
              port:
                number: 80

Use the nginx.ingress.kubernetes.io/use-regex: "true" for redirecting your static traffic to the correct service.

The above snippet will provide you the results as below:

1. Any request on xx.xx.com/status/ will be routed to one-service.
   More Example :
   xx.xx.com/status/zenduty -> one-service
   xx.xx.com/status/zenduty/incident -> one-service
   xx.xx.com/status/anything -> one-service
2. Any request on xx.xx.com containing .png, .js, .css, .jpg, .gif or .jpeg will be routed to static-app-service
   Example :
   xx.xx.com/custom.js -> static-app-service
   xx.xx.com/status/custom.js -> static-app-service
   xx.xx.com/custom.css -> static-app-service
   xx.xx.com/abc/zen/duty/custom.css -> static-app-service
   xx.xx.com/status/custom.gif -> static-app-service
3. Any other request on xx.xx.com which doesn't contain (.png, .js, .css, .jpg, .gif or .jpeg) and do not start with (xx.xx.com/static/), will be routed to third-service
   Example:
   xx.xx.com/login -> third-service
   xx.xx.com/xyz -> third-service

If you want to add another header to the snippet you can add it. My suggestion will be to create a separate ingress object for static content.

Below is how you can separate the above snippet and add more headers on custom service levels.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sample-app-ingress
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/server-snippet: |
      expires 1M;
      add_header Cache-Control "public";
spec:
  ingressClassName: nginx
  rules:
  - host: xx.xx.com
      http:
        paths:
        - pathType: Prefix
          path: /.*(js|css|gif|jpe?g|png)
          backend:
            service:
              name: static-app-service
              port:
                number: 80

The below snippet is for other services which do not have to serve the static content and have custom headers according to their requirements.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sample-app-ingress
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/server-snippet: |
      add_header X-XSS-Protection "1";
      add_header Strict-Transport-Security "max-age=31536000;includeSubdomains; preload";
spec:
  ingressClassName: nginx
  rules:
  - host: xx.xx.com
      http:
        paths:
        - pathType: Prefix
          path: /status/.*
          backend:
            service:
              name: one-service
              port:
                number: 80
  - host: xx.xx.com
     http:
       paths:
        - pathType: Prefix
          path: /
          backend:
            service:
              name: third-service
              port:
                number: 80

The ingress class name here is ingressClassName: nginx, please change accordingly.

This is the ingress way to route to the correct service.

If you want to pass client IP to your backend service , Check this for more details : https://stackoverflow.com/questions/65345240/updating-ingress-nginx-controller-configmap-to-pass-client-ip-to-backend-service/71407668#71407668

For Custom error page : How to customize error pages served via the default backend of an nginx ingress controller?