Nginx split_clients not dividing traffic

Question

I have the following nginx.config file:

events {}

http {
    # ...
    # application version 1a 
    upstream version_1a {
        server localhost:8090;
    }

    # application version 1b
    upstream version_1b {
        server localhost:8091;
    }

    split_clients "${arg_token}" $appversion {
        50%     version_1a;
        50% version_1b;
    }

    server {
        # ...
        listen 7080;
        location / {
            proxy_set_header Host $host;
            proxy_pass http://$appversion;
        }
    }
}

I have two nodejs servers listening on port 8090 and 8091 and I am hitting the URL http://localhost:7080, my expectation here is the Nginx will randomly split the traffic to version_1a and version_1b upstream, but, all the traffic is going to version_1a. Any insight into why this might be happening?

(I want to have this configuration for the canary traffic)

Answer 1

Validate the variable you are using to split the traffic is set correctly, and the variable's value should be uniformly distributed else the traffic will not be split evenly.

Answer 2

If you know the interval between raw samples in every time series (aka scrape_interval in Prometheus world), then the following query can be used:

min_over_time(meta_metric[20m]) > 0 and count_over_time(meta_metric[20m]) > N

Where N must be substituted with the minimum number of raw samples per each time series in 20 minute lookbehind window, which must be taken into account. For example, if the interval between raw samples is 1 minute, then N can be substituted with a number close to 20.