'Single get request on frontend returning an entire table in chunks - how to do? Django

Could use some help with this please.

Django and PostgreSQL is the backend for a React frontend project of mine. The frontend needs to fetch and perform calculations on a large number of rows in a table... a million + rows is possible.

Right now, fetching and doing those calculations is implemented, but it takes way too long, and that's locally. When deployed, it doesn't even work because it's not in batches, it's just one big GET request which times out after 30 seconds.

I've set up splicing on the frontend for dealing with POST/PUT related requests to get around that time out issue, but how would I go about doing a GET request in batches? I've been looking into iterator and cursor stuff to try and deal with things in chunks, but I am also confused - if a single GET request is performed on the frontend, is it even possible to send multiple responses? I have seen conflicting information on that.

Some guidance on the best way to address this would be appreciated. If it helps, I am primarily working with a class based way of doing things, Model, Serializer, ModelViewSet.

Here's what I'm working on currently to attempt to do batch get-requests, but not sure if I'm on the right page.

 def batch_get(self, request):
 count = UploadedShares.objects.all().count()
 chunk_size = 500
 for i in range(0, count, chunk_size):
         shares = UploadedShares.objects.all()[i:i + chunk_size]
         serialized_shares = serializers.serialize('json', shares)
        return JsonResponse(serialized_shares, safe=False)

Thanks.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source