Django + Postgres: OperationalError FATAL: sorry, too many clients already (as a result of a deliberate attack)

Question

There are many similar questions on Stackoverflow but they do not address my particular case. The problem is that there are deliberate attacks on the website with many simultaneous requests. I have mimicked these attacks running the following command:

ab -k -c 150 -n 90000 'https://www.mywebsite.com/'

The website gets down with the following error:

OperationalError at /
FATAL:  sorry, too many clients already

The number of max connections allowed on my Postgres instance is 100. Raising it, as recommended in most answers on Stackoverflow, is pointless, as there might be significantly higher numbers of requests during the deliberate attacks which we have been suffering lately.

I have tried remedying the issue by creating a special middleware caching the number of requests:

from ipware import get_client_ip

class IPMiddleware:
    REQUEST_PERIOD_IN_SECONDS = 10
    MAX_REQUESTS_IN_PERIOD = 100
    TOO_MANY_REQUESTS_BAN_PERIOD_IN_SECONDS = 60 * 2

    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        try:
            ip, is_routable = get_client_ip(request)
            if ip:

                if cache.get(f'ip_is_banned_{ip}', False):
                    raise PermissionDenied()

                cache_key = f'requests_made_{ip}'
                try:
                    cache.incr(cache_key)
                except ValueError:
                    cache.set(cache_key, 1, self.REQUEST_PERIOD_IN_SECONDS)

                requests_made = cache.get(cache_key)
                if requests_made > self.MAX_REQUESTS_IN_PERIOD:
                    cache.set(f'ip_is_banned_{ip}', True, self.TOO_MANY_REQUESTS_BAN_PERIOD_IN_SECONDS)

        except:
            pass

        response = self.get_response(request)
        return response

The order of my middlewares in settings.py is the following:

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'mywebsite.middlewares.CustomLocaleMiddleware',
    'mywebsite.middlewares.IPMiddleware',
    'maintenancemode.middleware.MaintenanceModeMiddleware',
]

The idea is to ban an IP that has made more than a number of requests within a period of time (e.g. 100 requests in 10 seconds). Then, if the IP is banned, PermissionDenied() is raised.

The cache logic works and the IP is getting banned. However, this does not prevent the simultaneous requests from hitting the database. I have tested it by changing the number of requests in a/b testing: it works fine with 50 requests but breaks the site with 150 requests.

Is there any efficient way to counteract such attacks except using Cloudflare?

Answer 1

Do you use nginx? Then you better limit the number of queries from the same ip using nginx. Django is too slow for that, imho. https://www.nginx.com/blog/rate-limiting-nginx/