How to improve performance of Django ORM already querying with prefetch_related? (many different tables involved)

Question

I have a Django model which is linked by many other models in a reverse ForeignKey relationship, that is many-to-one.

Illustration:

class ModelA:
    photo_url = URLField(null=True)


class AbstractModel:
    class Meta:
        abstract = True

    modelA_ref = ForeignKey(to=ModelA, on_delete=dj_models.CASCADE)


class ModelB(AbstractModel):
    whatever = TextField(null=True)


class ModelC(AbstractModel):
    whatever = TextField(null=True)


class ModelD(AbstractModel):
    whatever = TextField(null=True)

and so on for a dozen more.

I have a serializer for ModelA, and a corresponding view for when I call the corresponding endpoint /api/v1/modela/<uuid:key>/ in Django REST Framework.

Illustration:

from rest_framework.generics import RetrieveAPIView


class ModelASerializer:
    modelb_field = ListSerializer(source="modelb_set", child=ModelBSerializer())
    modelc_field = ListSerializer(source="modelc_set", child=ModelCSerializer())
    modeld_field = ListSerializer(source="modeld_set", child=ModelDSerializer())


class ModelAView(RetrieveAPIView):
    serializer_class = ModelASerializer
    lookup_field = "id"
    lookup_url_kwarg = "key"

    def get_queryset(self):
        key = self.kwargs["key"]

        return ModelA.objects.filter(id=key).prefetch_related(
            'modelb_set', 'modelc_set', 'modeld_set'
        )

Of course, this optimization does not prevent Django from querying all the different tables when the /api/v1/modela/<uuid:key>/ endpoint is reached, as expected: in my situation, the accumulation of queries takes 60% of the 1000ms response time, which is way too long.

Hence my question: is refactoring my database schema to make those models fit into one single table my only option? or is there something more elegant to be done?

Thank you for your time.