Django high memory usage

Question

I'm using django as a backend for a React frontend, and deploying both applications on Heroku.

I also use Gunicorn do serve the application and signed the Hobby plan on Heroku which offers 512 MB of RAM for the application to run.

But the django dyno, is almost always using a lot of memory, and exceeding the 512 mb limit. It goes down to only 40 MB of usage whenever I restart or deploy changes, but as soon as any user uses the system and calls some queries. The memory goes up a lot.

I've read about django and django-rest-framework memory optimization for some days now, and tried some changes like: using --preload on Gunicorn, setting --max-requests to kill process when they're too heavy on memory, I've also set CONN_MAX_AGE for the database and WEB_CONCURRENCY as stated on:

https://devcenter.heroku.com/articles/python-concurrency-and-database-connections

But none of that gave me a good enough result. What I'm guessing is wrong now are my queries, because I've seen some articles about the usage of .iterator() on queries and how it prevents de queries from being cached by the application and I didn't use it in any of my queries.

I don't think caching the queries would help on my application at all, I even store some of the results on React state exactly to keep the queries from being called again.

I tried using .iterator() on some queries but I noticed that when the memory goes up on the container it stays up for a very long time. I saw consumption remain the same for up to 6 hours straight, and I don't think that a query cache would be maintained in memory for so long (or would it?).

So, now I'm a little confused about what to try next and on what I should focus and any help is welcome. Thanks in advance!

EDIT

Just attached an image which shows the memory usage going up 60 MB only because I called the logout function!! Makes no sense to me... Also after it goes up it takes a really, really long time to go back down again. PRINT_FROM_HEROKU_LOGS

Answer 1

You have to use a memroy profiler to see what function or method allocate memory
An example tool is memray, after installing it, run the django server like this:

python -m memray run ./manage.py runserver

Visit the pages or call the APIs that might use a lot of memory then end the program run (on linux use CTRL+c)

It will generate a file with memory usage details and show you how to convert it to readable format, you can paste here to get some insights if you can't read it by yourself