large_client_header_buffers not working as fix for nginx 414 error

Question

I have a service which gets analytics data from multiple sites.

Now we can't send a POST request as browser stops the request when the tab is closed or switched, so we use a GET request inside an img tag which is sent to the server even if window is closed.

Since it's a get request, the request structure looks like this http://localhost/log?event=test&data=base64EncodedJSON

Now, if the JSON is big, the base64EncodedJSON will also be big.

If the URL is big, nginx returns a 414 error.

After research, I found out that large_client_header_buffers should be modified to allow larger url's, but for some reason, it doesn't seem to work.

Here's the nginx configuration

nginx.conf
user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    # Tried Fix for 414 error
    fastcgi_buffers 8 16k;
    fastcgi_buffer_size 32k;

    client_max_body_size 24M;
    client_body_buffer_size 128k;

    client_header_buffer_size 5120k;
    large_client_header_buffers 16 5120k;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

conf.d/default.conf

server {
        listen 80;
        #listen 443 ssl;
        #ssl_certificate /etc/nginx/ssl/server.pem;
        #ssl_certificate_key /etc/nginx/ssl/server.key;
        server_name _; #TODO: restrict server domain
        root /var/www/html/;

        location / {
            proxy_pass http://logger:8888/;
            proxy_cache off;
            proxy_set_header Host \$host;
            proxy_set_header X-Real-IP \$remote_addr;
        }
}

But even after setting large_client_header_buffers = 5120k, I still get a 414 error for big URLS.

Answer 1

Probably the fastcgi_buffers and fastcgi_buffer_size have to be adjusted accordingly to the large_client_header_buffers.
Additionally it's likely advisable to keep the values as small as possible. Especially for testing I'd increase slowly and check if the configured size is transferred--if not, check the exact size and compare it to configuration options that could be related.

Beside that the client_header_buffer_size can be kept small like 1k or 2k, if the length is larger the large_client_header_buffers kicks in.

Answer 2

Are you add this data to real-time Database if not then first add that data using model like UserModel(). Than use that code to insert it into realtime database

DatabaseReference  mDatabase = FirebaseDatabase.getInstance().getReference();

Here is user model

public class User {

public String username;
public String email;

public User() {
    
}

public User(String username, String email) {
    this.username = username;
    this.email = email;
}

}

Than use this code to insert it into real time db

mDatabase.child("users").child(userId).setValue(user);

Go to firebase real-time db and check is you data inserted. if inserted than use that code to read data

mDatabase.child("users").child(userId).get().addOnCompleteListener(new OnCompleteListener<DataSnapshot>() {
@Override
public void onComplete(@NonNull Task<DataSnapshot> task) {
    if (!task.isSuccessful()) {
        Log.e("firebase", "Error getting data", task.getException());
    }
    else {
        Log.d("firebase", String.valueOf(task.getResult().getValue()));
    }
}

});