Why is NextJS returning a 404 for certain sourcemap requests?

Question

Our NextJS deployment is currently returning a 404 for certain GET requests under /_next/static/. We're certain that this is due to some state in the JS process, because the 404s are non-deterministic, tend towards 1/instances, and stop when we scale down to one (1) NextJS instance.

After attempting the fixes described below, I'm stumped at why this could be happening.

Is anyone aware of what we might be doing wrong?

What we've tried

• Set Keep-Alive to true for all client requests to the NextJS server (though not NextJS server -> backend server requests). This is via fetch args.

  • We did this in the hope that the requests would all be routed to the same (Kubernetes) pod/process. I'm not 100% sure that the TLS connection is ultimately held by the pod, though (it's possible that some AWS/K8s layer is proxying at L6).

  • I'm aware that this may be invalidated by HTTP/2, which is being used (at least where available, inc/ on our browsers - point is, it's preferentially used on the server side).

• Set a constant Build ID, constant over all builds running at the same commit. (To be specific, it reads the contents of .git/refs/heads/master, and it's an invariant that we run at the latest head of master.)

  • We did this because it was mentioned in a GitHub thread somewhere - essentially an application of shotgun debugging. It had no perceptible effect.

Reproducing

This cURL request replicates the request made by my browser (with cookies removed due to length), with our root domain redacted since our company is not comfortable sharing it:

curl 'https://staging-subscriptions.xxxxxxxx.com/_next/static/chunks/main-3dff15b90b1a91be7a70.js' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:98.0) Gecko/20100101 Firefox/98.0' -H 'Accept: */*' -H 'Accept-Language: en-GB,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Connection: keep-alive' -H 'Referer: https://staging-subscriptions.xxxxxxxx.com/' -H 'Sec-Fetch-Dest: script' -H 'Sec-Fetch-Mode: no-cors' -H 'Sec-Fetch-Site: same-origin' -H 'TE: trailers' -v -o/dev/null

The relevant output is:

* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.xxxxxxxx.com
*  start date: Feb  3 00:00:00 2022 GMT
*  expire date: Mar  6 23:59:59 2023 GMT
*  subjectAltName: host "staging-subscriptions.xxxxxxxx.com" matched cert's "*.xxxxxxxx.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x11f811000)
> GET /_next/static/chunks/main-3dff15b90b1a91be7a70.js HTTP/2
> Host: staging-subscriptions.xxxxxxxx.com
> user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:98.0) Gecko/20100101 Firefox/98.0
> accept: */*
> accept-language: en-GB,en;q=0.5
> accept-encoding: gzip, deflate, br
> connection: keep-alive
> referer: https://staging-subscriptions.xxxxxxxx.com/
> sec-fetch-dest: script
> sec-fetch-mode: no-cors
> sec-fetch-site: same-origin
> te: trailers
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 404 
< date: Fri, 08 Apr 2022 18:00:06 GMT
< content-type: text/html; charset=utf-8
< x-dns-prefetch-control: off
< x-frame-options: SAMEORIGIN
< strict-transport-security: max-age=15552000; includeSubDomains
< x-download-options: noopen
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< cache-control: private, no-cache, no-store, max-age=0, must-revalidate
< x-powered-by: Next.js
< etag: "1c1e-21u1PU3UIdT361B/fc2BkKjKzrA"
< vary: Accept-Encoding
< content-encoding: gzip
< 
{ [10 bytes data]
100  2724    0  2724    0     0   6328      0 --:--:-- --:--:-- --:--:--  6394
* Connection #0 to host staging-subscriptions.xxxxxxxx.com left intact