Nginx HTTPS proxy_pass every path

Question

I'm trying to setup https using nginx and cloudflare. I generated a certificate and key using cloudflare and added them to my nginx config (See below).

server {
    listen 443 ssl;
    server_name <URL_HIDDEN>;

    ssl_certificate /etc/nginx/own-certs/server.crt;
    ssl_certificate_key /etc/nginx/own-certs/server.key;

    location / {
        proxy_pass http://localhost:8082;
        proxy_http_version 1.1;
        proxy_set_header UPgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

The problem: Only the main page works (/). When going to any other page (Like /favicon.ico), I'm getting a 504 error after 30 seconds. When using the specific port and same path (http://localhost:8082/favicon.ico), everything works fine. Everything works fine when I remove the ssl part too.

It isn't a nginx timeout issue, because I'm getting a response within 5ms when using the localhost URL.