Enabling bidirectional UDP communication across an NGINX loadbalancer in Kubernetes

Question

I'm working on a project where I'm hoping to host a series of servers on Kubernetes pods that use UDP to communicate between the client and server. One pod per service, one service per game server. To publicly expose them with a single IP, I'm using an nginx L4 loadbalancer backed by AWS. Let's look at one such service, which listens and responds on ports 7777 (gameplay) and 27015 (Steam query port).

Although UDP requests are reaching my services fine, my services are sending UDP messages back that aren't making it to the client. The steam query port however sends a response that is received, likely because proxy_responses is set to 1 by default in nginx.conf, and this steam query has a 1:1 request:response pattern for a player to query an address's servers. However when the client interacts with the game server to connect and play, there seems to a more sporadic or patternless bidirectional UDP conversation where the server will send one, many, or no UDP messages pack periodically. logs from the controller pod when attempting game connection show "upstream timed out (110: Operation timed out) while proxying connection".

When I set proxy_responses to 0 in nginx.conf to eliminate upstream response expectations, even the steam query response doesn't get through. "tcpdump -nei eth0 udp port 27015" on the service shows incoming messages and replies to the loadbalancer from the service:

08:59:01.167167 36:8e:ae:75:a3:02 > 26:30:b0:fc:06:f2, ethertype IPv4 (0x0800), length 67: 172.31.64.95.37278 > 172.31.64.143.27015: UDP, length 25
08:59:01.184319 26:30:b0:fc:06:f2 > 36:8e:ae:75:a3:02, ethertype IPv4 (0x0800), length 291: 172.31.64.143.27015 > 172.31.64.95.37278: UDP, length 249
08:59:18.370711 36:8e:ae:75:a3:02 > 26:30:b0:fc:06:f2, ethertype IPv4 (0x0800), length 67: 172.31.64.95.40092 > 172.31.64.143.27015: UDP, length 25
08:59:18.371635 26:30:b0:fc:06:f2 > 36:8e:ae:75:a3:02, ethertype IPv4 (0x0800), length 291: 172.31.64.143.27015 > 172.31.64.95.40092: UDP, length 249

The logs on the controller pod are just several lines of "UDP 200 0 25 0.000"

It seems what I'm looking for is an bidirectional UDP line of communication between the game pod and the client across this NGINX loadbalancer without a specific request/response pattern, but the upstream responses aren't getting through the loadbalancer. Any suggestions?

---

I used this yaml as a template for my nginx deployment. I modified the .spec.ports field on the ingress-nginx-controller service in my helm template as follows:

ports:
  {{- range $port := .Values.ports }}
  - port: {{ $port.number }}
    targetPort: {{ $port.number }}
    protocol: {{ $port.protocol }}
    name: {{ $port.name | replace "_" "-" | lower }}
  {{- end }}

The port number here resolves to each port I want to expose and the protocol resolves to UDP. I added a similar section to the .spec.template.spec.containers.ports section of the ingress-nginx-controller deployment:

{{- range $port := .Values.ports }}
- name: {{ print "game" "-" $port.idx  }}
  containerPort: {{ $port.number }}
  protocol: {{ $port.protocol }}
{{- end }}

I also added the following configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: udp-services
  namespace: ingress-nginx
data:
  {{- range $port := .Values.ports }}
  {{ $port.number }}: {{ print "default/" "service-" $.Values.env "-" $port.game "-" $port.map ":" $port.number | lower }}
  {{- end }}

The name of the namespace/service aligns with my respective game services. As do the ports, though there are multiple per service. I added the companion arg flag to the ingress-nginx-controller deployment's container args (along with the preexisting ones):

- --udp-services-configmap=$(POD_NAMESPACE)/udp-services

An example of as stream block in the nginx.conf is as follows:

server {
    preread_by_lua_block {
        ngx.var.proxy_upstream_name="udp-default-service-dev-game-map-7777";
    }
    
    listen                  7777 udp;
    
    proxy_responses         0;
    proxy_timeout           600s;
    proxy_next_upstream     on;
    proxy_next_upstream_timeout 600s;
    proxy_next_upstream_tries   3;
    proxy_pass              upstream_balancer;
}

upstream_balancer appears to be a dummy placeholder value. Imagine the same block with a different port for 27015.

As stated above before the break, does anyone know how to fix the issue of UDP responses from upstream not getting through NGINX?